Vehicle-mounted panoramic image generation method, computer device, computer storage medium, computer program product and mobile platform

Abstract

Disclosed in the present application are a vehicle-mounted panoramic image generation method, a computer device, a computer storage medium, a computer program product and a mobile platform. The vehicle-mounted panoramic image generation method comprises: acquiring in real time multiple pieces of image data collected by a plurality of cameras; determining point cloud data corresponding to the multiple pieces of image data; on the basis of the point cloud data, correcting a preset three-dimensional model to obtain a corrected three-dimensional model; on the basis of the multiple pieces of image data, performing texture mapping on the corrected three-dimensional model to obtain a dynamic real-scene model; and on the basis of vehicle information and the dynamic real-scene model, generating a vehicle-mounted panoramic image. In the present application, point cloud data corresponding to multiple pieces of image data acquired in real time is used, a preset three-dimensional model is corrected on the basis of the point cloud data, and the corrected three-dimensional model is used to perform texture mapping to obtain a dynamic real-scene model, thereby solving the problem in the prior art whereby ambient objects with height are deformed and stretched due to a mismatch between a preset three-dimensional model and object shapes.

Description

Methods for generating panoramic images of vehicles, computer equipment, computer storage media, computer program products, and mobile platforms Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method, apparatus, computer equipment, computer storage medium, computer program product, and mobile platform for generating in-vehicle panoramic images. Background Technology

[0002] 360° surround-view imaging for vehicles is gradually becoming a research hotspot in the automotive field. It typically involves stitching together multiple images output from surround-view cameras distributed around the vehicle to generate a panoramic image that surrounds the vehicle. Most in-vehicle panoramic imaging systems only support bird's-eye view or a single view from a specific camera angle, but some high-end models are gradually beginning to support 3D scene rendering.

[0003] Because different fisheye cameras are typically mounted around the vehicle body, it is difficult to reconstruct an accurate unified field coordinate system in a real-time system. Therefore, traditional 3D stitching methods pre-generate a complete, closed "bowl-shaped" model to perform direct 3D point-to-fisheye image mapping calculations. Consequently, in scenes with tall objects around the vehicle body, traditional pseudo-3D panoramic stitching algorithms exhibit significant stretching and deformation. Summary of the Invention

[0004] This application provides a method, apparatus, computer device, computer storage medium, computer program product, and mobile platform for generating in-vehicle panoramic images, which are used to solve at least one of the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this application provide a method for generating in-vehicle panoramic images, comprising: acquiring multi-channel image data collected by multiple cameras in real time; determining point cloud data corresponding to the multi-channel image data; correcting a preset three-dimensional model based on the point cloud data to obtain a corrected three-dimensional model; performing texture mapping on the corrected three-dimensional model based on the multi-channel image data to obtain a dynamic real-scene model; and generating an in-vehicle panoramic image based on vehicle information and the dynamic real-scene model.

[0006] In some embodiments, determining the point cloud data corresponding to the plurality of road image data includes: filtering valid pixels in the plurality of road image data; and determining the point cloud data of objects around the vehicle body based on the coordinate information of the valid pixels.

[0007] In some embodiments, filtering valid pixels in the multi-channel image data includes: determining depth estimates of pixels in the multi-channel image data based on a depth estimation network; and filtering pixels whose depth estimates are within a preset threshold range to determine the valid pixels.

[0008] In some embodiments, filtering valid pixels in the multi-channel image data includes: determining the depth estimate of pixels in the multi-channel image data according to a depth estimation network; filtering pixels whose depth estimate is within a preset threshold range; and deleting abruptly changing pixels with abruptly changing depth values ​​among the pixels whose depth estimate is within the preset threshold range, thereby determining the valid pixels.

[0009] In some embodiments, deleting mutated pixels with abrupt changes in depth value among pixels whose depth estimates are within a preset threshold range includes: traversing pixels whose depth estimates are within a preset threshold range according to a preset neighborhood range to identify mutated pixels with abrupt changes in depth value.

[0010] In some embodiments, filtering effective pixels in the multi-channel image data includes: determining the depth estimate of pixels in the multi-channel image data according to a depth estimation network; filtering pixels whose depth estimate is within a preset threshold range; deleting abruptly changed pixels whose depth estimate is within the preset threshold range; and performing pixel fusion to determine the effective pixels by dividing the point cloud space into voxels for the pixels after deleting abruptly changed pixels.

[0011] In some embodiments, the point cloud space of the pixels after removing the mutated pixels is divided into voxels, and pixel fusion is performed to determine the effective pixels, including: dividing the point cloud space of the pixels after removing the mutated pixels into voxels; for pixels falling within the same voxel unit, the center point of the same voxel unit is used as the fusion result to determine the effective pixels.

[0012] In some embodiments, correcting a preset 3D model based on the point cloud data to obtain a corrected 3D model includes: traversing the 3D points of the preset 3D model based on the point cloud data; and fusing pixels in the point cloud data belonging to the same preset range with the 3D points on the preset 3D model.

[0013] In some embodiments, the method further includes: deleting 3D points that have not been fused after traversing the 3D points of the preset 3D model using multiple sets of the point cloud data from the preset 3D model; wherein the multiple sets of the point cloud data correspond to multiple sets of the multi-channel image data.

[0014] In some embodiments, the multi-camera system includes an adjacent first camera and a second camera, with an overlapping visible area between the first camera and the second camera. The method further includes: for overlapping pixels located within the overlapping visible area, obtaining a first pixel value corresponding to the first camera and a second pixel value corresponding to the second camera; and determining the pixel value of the overlapping pixel based on the first pixel value and the second pixel value.

[0015] In some embodiments, the method further includes: determining a first weighted value corresponding to the first camera and a second weighted value corresponding to a majority of second cameras based on the distance of the overlapping pixel from the boundary of the overlapping visible area; determining the pixel value of the overlapping pixel based on the first pixel value and the second pixel value, including: performing a weighted summation of the first pixel value and the second pixel value using the first weighted value and the second weighted value to determine the pixel value of the overlapping pixel.

[0016] Secondly, this application also provides an in-vehicle panoramic image generation device, comprising:

[0017] The data acquisition module is used to acquire multi-channel image data from multiple cameras in real time.

[0018] The point cloud data determination module is used to determine the point cloud data corresponding to the multi-channel image data;

[0019] The model correction module is used to correct the preset 3D model based on the point cloud data to obtain the corrected 3D model;

[0020] The texture mapping module is used to perform texture mapping on the corrected 3D model based on the multi-channel image data to obtain a dynamic real-world model.

[0021] The panoramic image generation module is used to generate in-vehicle panoramic images based on vehicle information and the dynamic real-scene model.

[0022] Thirdly, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the vehicle panoramic image generation method described in any embodiment of this application.

[0023] Fourthly, this application also provides a computer-readable storage medium storing a computer program / instructions thereon, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the vehicle panoramic image generation method described in any embodiment of this application.

[0024] Fifthly, this application also provides a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the vehicle panoramic image generation method described in any embodiment of this application.

[0025] Sixthly, this application also provides a mobile platform, characterized in that it is equipped with the computer device described in any embodiment of the application.

[0026] This application uses point cloud data from real-time acquired multi-channel image data to obtain the real distribution of objects around the vehicle. Based on this, the preset 3D model is corrected, and then the corrected 3D model is used for texture mapping to obtain a dynamic real-scene model. This solves the problem in the prior art where objects of a certain height in the surrounding environment are deformed and stretched due to the mismatch between the preset 3D model and the object shape. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 is a flowchart of an embodiment of the vehicle panoramic image generation method of this application;

[0029] Figure 2 is a flowchart of another embodiment of the vehicle panoramic image generation method of this application;

[0030] Figure 3 is a flowchart of another embodiment of the vehicle panoramic image generation method of this application;

[0031] Figure 4 is a flowchart of another embodiment of the vehicle panoramic image generation method of this application;

[0032] Figure 5 is a flowchart of another embodiment of the vehicle panoramic image generation method of this application;

[0033] Figure 6 is a flowchart of another embodiment of the vehicle panoramic image generation method of this application;

[0034] Figure 7 is a schematic diagram of deleting abruptly changed pixels within a 9-neighborhood in this application;

[0035] Figure 8 is a schematic diagram of the area division of the shooting angle range of the multi-channel fisheye camera projected onto the horizontal plane in this application;

[0036] Figure 9 is a flowchart of another embodiment of the vehicle panoramic image generation method of this application;

[0037] Figure 10 is a schematic diagram of the structure of an embodiment of the computer device of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0039] It should also be noted that, in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0040] This application provides a method for generating in-vehicle panoramic images, which can be used in the vehicle infotainment system of autonomous or assisted driving vehicles to achieve panoramic presentation of the vehicle's surroundings.

[0041] As shown in Figure 1, an embodiment of this application provides a method for generating in-vehicle panoramic images, including:

[0042] S10: Real-time acquisition of multi-channel image data from multiple cameras.

[0043] For example, the multi-camera system consists of four monocular fisheye cameras, distributed around the vehicle. These four cameras acquire real-time environmental information about the vehicle's surroundings, generating four channels of image data.

[0044] S20. Determine the point cloud data corresponding to the multi-channel image data.

[0045] For example, point cloud data corresponding to multiple image data streams are determined separately. Using four image data streams, four point cloud data streams are determined accordingly.

[0046] For each frame of multi-channel image data input from four monocular fisheye cameras, the normalized camera coordinates (x, y, 1) of each pixel are calculated using the intrinsic parameter matrices of each camera, as follows: (x, y, 1) = K(u, v, 1), where K is the camera's intrinsic parameter matrix, and u and v are the pixel's horizontal and vertical coordinates in the pixel coordinate system. The pixel coordinate system refers to the fact that an image is composed of pixels, and each pixel can be represented by an ordered pair (u, v). This coordinate system is called the pixel coordinate system, a two-dimensional coordinate system that identifies the position of each pixel in the image sensor, with each increment being one pixel. The intrinsic parameter matrices of the four monocular fisheye cameras are calibrated as needed. Further, based on the normalized camera coordinates (x, y, 1) of each pixel, the 3D coordinates (X, Y, Z) of each pixel are calculated. The calculation formula is: (X, Y, Z) = d*(x, y, 1), where d is the depth value of the pixel.

[0047] For example, the depth value d of a pixel can be, but is not limited to, the following:

[0048] Method 1. Obtain the depth value of the pixel using sensors such as LiDAR or structured light.

[0049] Method 2. Calculate the depth value of a pixel using the parallax information from a binocular or multi-view camera.

[0050] Method 3. Analyze the image using prior knowledge, models, or depth estimation networks to infer the depth value of each pixel.

[0051] In some embodiments of this application, the depth estimation network in Method 3 is used to determine the depth value of a pixel, which will be described in a later section.

[0052] S30. The preset three-dimensional model is corrected based on the point cloud data to obtain the corrected three-dimensional model.

[0053] The preset 3D model is a model pre-constructed as needed, and can be constructed using relevant existing technologies; this application does not limit its scope. For example, the preset 3D model can be implemented as a preset bowl-shaped model. Based on the vehicle body size parameters, a "bowl-shaped" model covering a certain area around the vehicle body is generated. The radius of the bowl's bottom and the height of the bowl's walls are adjusted according to the user's field of vision requirements (e.g., the radius of the bowl's bottom is set to 12m, and the height of the bowl's walls is set to 3.5m).

[0054] For example, the three-dimensional points of the preset bowl-shaped model are traversed according to the point cloud data; pixels in the point cloud data and three-dimensional points on the preset bowl-shaped model that belong to the same preset range are fused. For example, the point cloud distribution of objects around the vehicle body in the current frame is obtained based on all valid pixels of multi-channel image data (e.g., multi-channel surround view images), the three-dimensional points of the preset bowl-shaped model are traversed, and points within the preset range of the three-dimensional points of the preset bowl-shaped model and the 3D points in the monocular point cloud (e.g., the three-dimensional points of the preset bowl-shaped model and the 3D points in the monocular point cloud fall within the same voxel unit) are fused, and the coordinates of the fused point are taken as the center coordinates of the voxel unit.

[0055] S40. Based on the multi-channel image data, perform texture mapping on the corrected 3D model to obtain a dynamic real-world model.

[0056] For example, texture mapping includes calculating the pixel coordinates corresponding to each 3D point using the formula (u,v)=K(R(X,Y,Z)+t), where K is the intrinsic parameter matrix, and R and t are the extrinsic parameter rotation matrix and translation coefficient.

[0057] S50. Generate an in-vehicle panoramic image based on the vehicle information and the dynamic reality model. The vehicle information includes, but is not limited to, the vehicle's size and position information. For example, the vehicle's relative position in the dynamic reality model is determined based on its position information, and its boundary information in the dynamic reality model is determined based on its size information, thereby embedding the vehicle into the dynamic reality model to generate the in-vehicle panoramic image. Existing technologies can be used to generate in-vehicle panoramic images based on vehicle information and the dynamic reality model; this application does not limit this method.

[0058] This embodiment uses point cloud data from real-time acquired multi-channel image data (i.e., obtains the real object distribution around the vehicle) to correct the preset 3D model. Then, the corrected 3D model is used for texture mapping to obtain a dynamic real-scene model, which solves the problem in the prior art where objects of height in the surrounding environment are deformed and stretched due to the mismatch between the preset 3D model and the object shape.

[0059] Figure 2 shows a flowchart of an embodiment of the vehicle-mounted panoramic image generation method of this application. In this embodiment, determining the point cloud data corresponding to the plurality of road image data includes:

[0060] S21. Screen the valid pixels in the multi-channel image data. Exemplarily, first perform depth estimation on the pixels in the multi-channel image data to obtain the depth estimation value of each pixel, and then screen out the valid pixels according to the depth estimation value of each pixel. Here, depth estimation refers to obtaining the distance from each pixel in the scene of the image to the camera. Exemplarily, perform depth estimation according to a depth estimation network (as an example, in this application, the depth estimation network can be but is not limited to: ZoeD-M12-NK) to obtain the depth estimation value d of each pixel of each frame of the surround-view image in the multi-channel image data, and preliminarily screen whether the depth estimation value of each pixel is valid according to the following criterion, and determine the pixel corresponding to the valid depth estimation value as a valid pixel. The screening criterion is: min_d < d < max_d, delete the pixels with depth values greater than the maximum threshold max_d and less than the minimum threshold min_d. Optionally, the value of min_d is 0.1 m, and the value of max_d is 6.5 m.

[0061] S22. Determine the point cloud data of the objects around the vehicle body according to the coordinate information of the valid pixels. Exemplarily, determine the 3D point coordinates (X, Y, Z) of the valid pixels according to the coordinate information (u, v) of the valid pixels, and determine the point cloud data of the objects around the vehicle body based on the 3D point coordinates of the valid pixels.

[0062] For example, as introduced in the foregoing embodiment: Combine the internal parameter matrix of each monocular fisheye camera to calculate the normalized camera coordinate system coordinates (x, y, 1) of each pixel. The formula is as follows: (x, y, 1) = K(u, v, 1), where K is the internal parameter matrix of the camera; Further, according to the normalized camera coordinate system coordinates (x, y, 1) of each pixel, calculate the 3D point coordinates (X, Y, Z) of each pixel. Here, the calculation formula is: (X, Y, Z) = d * (x, y, 1). Finally, use the 3D coordinates of multiple valid pixels of the objects around the vehicle body as the point cloud data of the objects around the vehicle body.

[0063] In this embodiment, by screening the valid pixels in the multi-channel image data to determine the point cloud data of the objects around the vehicle body, it avoids the influence of useless pixels on the correction of the preset three-dimensional model, improves the performance of the corrected three-dimensional model, and ensures the quality of the generated in-vehicle 3D image.

[0064] As shown in FIG. 3, it is a flowchart of an embodiment of the method for generating an in-vehicle panoramic image of the present application. In this embodiment, screening the valid pixels in the multi-channel image data includes:

[0065] S211. Determine the depth estimation value of the pixels in the multi-channel image data according to the depth estimation network.

[0066] Exemplarily, a depth estimation value d of each pixel of each frame of环视影像in the multi-channel image data is obtained according to the depth estimation network.

[0067] S212. Screen pixels whose depth estimation values are within a preset threshold range for determining the valid pixels. Exemplarily, the following criteria are used to preliminarily screen whether the pixel depth is valid. The screening criteria are: min_d < d < max_d, and pixels with depth values greater than the maximum threshold max_d and less than the minimum threshold min_d are deleted.

[0068] In this embodiment, by determining whether the depth estimation value of a pixel is within a preset range, invalid pixels with too large or too small depth estimation values are screened out. This avoids the influence of useless pixels on the correction of the preset 3D model, improves the performance of the corrected 3D model, and ensures the quality of the generated vehicle-mounted 3D image.

[0069] As shown in FIG. 4, it is a flowchart of an embodiment of the vehicle-mounted panoramic image generation method of the present application. In this embodiment, screening the valid pixels in the multi-channel image data includes:

[0070] S211. Determine the depth estimation value of the pixels in the multi-channel image data according to the depth estimation network.

[0071] Exemplarily, a depth estimation value d of each pixel of each frame of环视影像in the multi-channel image data is obtained according to the depth estimation network.

[0072] S212. Screen pixels whose depth estimation values are within a preset threshold range.

[0073] Exemplarily, the following criteria are used to preliminarily screen whether the pixel depth is valid. The screening criteria are: min_d < d < max_d, and pixels with depth values greater than the maximum threshold max_d and less than the minimum threshold min_d are deleted.

[0074] S213. Delete the mutant pixels with mutated depth values among the pixels whose depth estimation values are within the preset threshold range (i.e., the pixels obtained by preliminary screening based on the depth estimation values) to determine the valid pixels.

[0075] Exemplarily, traverse the pixels whose depth estimation values are within the preset threshold range according to a preset neighborhood range to determine the mutant pixels with mutated depth values; delete the mutant pixels from the pixels whose depth estimation values are within the preset threshold range.

[0076] In this embodiment, first, pixels whose depth estimation values are within the preset threshold range are screened out as a whole, and then the mutant pixels with mutated depth values are further screened out. This not only realizes the screening of valid pixels as a whole, but also screens out the mutant pixels with obvious abnormal mutations according to the mutation situation of the depth values, so as to improve the accuracy and reliability of the determined valid pixels.

[0077] In this embodiment, the preset neighborhood range is set to a 9-neighborhood range, and it is determined whether there are mutant pixels with depth mutations within the preset neighborhood range. For the depth results after preliminary screening (i.e., the pixels obtained by preliminary screening based on the depth estimation values), the outliers with depth mutations are further removed. The screening method is to compare the 9-neighborhood around the target pixel (where the target pixel corresponds to the central pixel within the 9-neighborhood), and the screening criterion is: |d - d n | > trd, where d is the depth estimation value of the target pixel, and d n is the depth estimation value of the nth pixel within the 9-neighborhood, n takes values from 1 to 7, and the threshold trd is preferably set to 0.3 m. When there are at least 7 pixels within the 9-neighborhood that satisfy |d - d n | > trd, the target pixel is considered a mutant outlier, and the target pixel is deleted. The 9-neighborhood range is a 3x3 range centered on the target pixel. The reason for setting the 9-neighborhood range is that the inventors found during the implementation of this application that too few neighborhood ranges have a large accidental error problem, and too large neighborhood ranges will reduce the algorithm performance. At the same time, the pixel similarity within the neighborhood is not enough (the range is too large), resulting in errors.

[0078] In this embodiment, first, pixels with depth estimation values within the preset threshold range are screened out as a whole. Further, by traversing according to the preset neighborhood range, mutant pixels with depth value mutations are locally screened out. This not only effectively screens valid pixels as a whole but also screens out mutant pixels with obvious abnormal mutations within the local range, improving the accuracy and reliability of the determined valid pixels, thereby achieving the purpose of real-time fitting the shape of the real object.

[0079] As shown in FIG. 5, it is a flowchart of an embodiment of the method for generating an in-vehicle panoramic image according to the present application. In this embodiment, screening the valid pixels in the multi-channel image data includes:

[0080] S211. Determine the depth estimation values of the pixels in the multi-channel image data according to the depth estimation network.

[0081] Exemplarily, according to the depth estimation network, the depth estimation value d of each pixel in each frame of the环视影像 (surround-view image) in the multi-channel image data is obtained.

[0082] S212. Screen out the pixels with depth estimation values within the preset threshold range.

[0083] Exemplarily, the pixels with depth estimation values within the preset threshold range are preliminarily screened according to the following criterion. The screening criterion is: min_d < d < max_d, and the pixels with depth values greater than the maximum threshold max_d and less than the minimum threshold min_d are deleted.

[0084] S213. Delete mutated pixels whose depth values ​​change abruptly among pixels whose depth estimates are within a preset threshold range. For example, traverse pixels whose depth estimates are within a preset threshold range according to a preset neighborhood range to identify mutated pixels whose depth values ​​change abruptly; delete the mutated pixels from the pixels whose depth estimates are within the preset threshold range.

[0085] S214. For the point cloud space of pixels after removing aberration pixels, voxels are divided, and pixel fusion is performed to determine the effective pixels. For example, the point cloud space of pixels after removing aberration pixels is divided into voxels; for pixels falling within the same voxel unit, the center point of the same voxel unit is used as the fusion result to determine the effective pixels. The point cloud space division of pixels after removing aberration pixels includes: dividing the point cloud space of pixels after removing aberration pixels into voxels of a predetermined size. The predetermined size can be 3-5 cm, that is, the point cloud space of pixels after removing aberration pixels can be divided into voxels according to a cube with a side length of 3-5 cm, and each voxel is a cube with a side length of 3-5 cm, containing multiple pixels. In this embodiment, the center point of the same voxel unit is used as the fusion result, filtering out redundant pixels in the same voxel unit to obtain effective pixels. This avoids the influence of redundant pixels in the same voxel unit on the correction of the preset 3D model, improving the realism of the corrected 3D model in 3D scene reconstruction.

[0086] In this embodiment, three-dimensional spatial voxel division and fusion of point clouds are used to obtain effective pixels, and the effective pixels are used to correct the preset three-dimensional model, thereby improving the realism of the 3D scene reconstruction of the corrected three-dimensional model.

[0087] In some embodiments, correcting a preset 3D model based on the point cloud data to obtain a corrected 3D model includes:

[0088] The three-dimensional points of the preset three-dimensional model are traversed based on the point cloud data.

[0089] Pixels in the point cloud data that belong to the same preset range and 3D points on the preset 3D model are fused together.

[0090] After traversing the three-dimensional points of the preset three-dimensional model using multiple sets of point cloud data, any three-dimensional points that are still not merged are deleted from the preset three-dimensional model; wherein, the multiple sets of point cloud data correspond to the multiple sets of multi-channel image data.

[0091] For example, the 3D points of a preset 3D model are traversed based on the point cloud data; pixels in the point cloud data and 3D points on the preset 3D model that belong to the same preset range are fused. For example, depth estimation can be performed based on all valid pixels of multi-channel image data (e.g., multi-channel surround view images) to obtain the point cloud distribution of objects around the vehicle body in the current frame. The 3D points of the preset 3D model are traversed, and points within the preset range of the 3D points of the preset 3D model and the 3D points in the monocular point cloud (e.g., the 3D points of the preset 3D model and the 3D points in the monocular point cloud fall within the same voxel unit) are fused, with the coordinates of the fused point taken as the center coordinates of the voxel unit.

[0092] For example, isolated points in a 3D model (points that have not been fused across multiple consecutive frames) are eliminated. Typically, in a 20Hz system, the value for "multiple frames" is 10 frames; for higher frame rate systems, this can be increased appropriately (e.g., 20 frames, 30 frames, etc.). Multiple frames reflect the lifecycle of a point in a 3D model. If, across multiple consecutive frames, some points in the point cloud data fail to fuse with points in the 3D model, then those points are considered dead or useless and need to be deleted.

[0093] Figure 6 shows a flowchart of an embodiment of the vehicle-mounted panoramic image generation method of this application. In this embodiment, the multi-camera system includes an adjacent first camera and a second camera, with an overlapping visible area between the first camera and the second camera. The vehicle-mounted panoramic image generation method further includes:

[0094] S60. For overlapping pixels located within the overlapping visible area, obtain a first pixel value corresponding to the first camera and a second pixel value corresponding to the second camera.

[0095] For example, the common visible area of ​​adjacent cameras is divided into a buffer (i.e., the overlapping area of ​​the shooting view range of adjacent cameras). For 3D points (i.e., overlapping pixels) within the buffer, two corresponding pixels on the images captured by the adjacent cameras are determined, and the pixel values ​​of the two pixels are obtained (e.g., a first pixel value and a second pixel value. For example, the first pixel value is the pixel value of the pixel on the image captured by the first camera, and the second pixel value is the pixel value of the pixel on the image captured by the second camera).

[0096] S70. Determine the pixel value of the overlapping pixel based on the first pixel value and the second pixel value.

[0097] In some embodiments, the vehicle-mounted panoramic image generation method of this application further includes determining a first weighted value corresponding to the first camera and a second weighted value corresponding to the second camera based on the distance of the overlapping pixels from the boundary of the overlapping visible area. Taking the first camera as an example, for overlapping pixels located within the overlapping visible area, the closer the overlapping pixel is to the boundary of the first camera's visible area (corresponding to the boundary farther from the first camera within the overlapping visible area boundary), the smaller the corresponding first weighted value and the larger the corresponding second weighted value; conversely, the larger the first weighted value, the smaller the corresponding second weighted value. In some embodiments, the sum of the first weighted value and the second weighted value is 1. Determining the pixel value of the overlapping pixel based on the first pixel value and the second pixel value includes: weighting and summing the first pixel value and the second pixel value using the first weighted value and the second weighted value to determine the pixel value of the overlapping pixel.

[0098] In some embodiments, the vehicle panoramic image generation method of this application further includes comparing a first distance between overlapping pixels and a first camera and a second distance between overlapping pixels and a second camera; when the first distance is less than the second distance, the first pixel value is determined as the pixel value of the overlapping pixel; otherwise, the second pixel value is determined as the pixel value of the overlapping pixel.

[0099] In some embodiments, the vehicle panoramic image generation method of this application further includes determining the first image quality of the image corresponding to the first pixel value and the second image quality of the image corresponding to the second pixel value; if the first image quality is better than the second image quality, then the first pixel value is determined as the pixel value of the overlapping pixel; otherwise, the second pixel value is determined as the pixel value of the overlapping pixel.

[0100] For example, the overlapping visible area is explained as follows: The shooting angle range of the multi-channel fisheye camera has an overlapping visible area in three-dimensional space. When the shooting angle range of the multi-channel fisheye camera is projected onto the horizontal plane, there is still an overlapping visible area between adjacent fisheye cameras, which is defined as a buffer zone.

[0101] Figure 8 shows a schematic diagram of the region division when projecting the shooting angle range of a multi-channel fisheye camera onto a horizontal plane. The gray area represents the non-buffer zone, and the blank sector-shaped areas represent the buffer zones. Let's take one of the buffer zones as an example. Assume the total central angle of this buffer zone's sector is θ. There is a point P within this sector. The line connecting point P to the sector's vertex makes angles α and (θ-α) with the two boundaries of the sector, respectively. The pixel values ​​corresponding to this point in the two fisheye images are A and B. Therefore, the final weighted pixel value of this point is: A*(α / θ) + B*(1-α / θ).

[0102] As shown in FIG. 9, in the vehicle-mounted panoramic image generation method provided by some embodiments of the present application, by using the results of the monocular depth estimation network to dynamically adjust the point distribution of the preset model in real time, the authenticity of 3D reconstruction can be improved. In this embodiment, taking the input of 4 fisheye cameras 1-4 as an example, images 1 to 4 are collected, and then real-time monocular depth estimation is performed based on images 1 to 4, and a monocular point cloud is generated based on the depth estimation results. Further, the pre-generated preset bowl-shaped model is corrected by using the generated monocular point cloud to obtain a corrected bowl-shaped model; then, images 1-4 are texture-mapped to the corrected bowl-shaped model, and the final real-scene 3D model is generated in combination with the pre-constructed vehicle model.

[0103] More specifically, the method proposed in the present application includes the following steps and processes. The following steps take the input of 4 fisheye cameras as an example:

[0104] 1. Generate a "bowl-shaped" model that covers a certain range around the vehicle body according to the vehicle body size parameters. The radius of the bottom of the bowl and the height of the bowl wall are adjusted according to the user's field of view requirements;

[0105] 2. For each frame of the multi-channel surround view images input by the sensor, in combination with the camera internal parameters, calculate the normalized camera coordinate system coordinates (x, y, 1) of each pixel. The formula is as follows: (x, y, 1) = K(u, v, 1), where K is the internal parameter matrix of the camera;

[0106] 3. Obtain the depth estimation value d of each pixel of each frame of the surround view image according to the depth estimation network, and initially screen whether the pixel depth is valid according to the following criteria; the screening criteria are: min_d < d < max_d, and pixels with depth values greater than the maximum threshold max_d and less than the minimum threshold min_d are deleted;

[0107] 4. For the depth results preliminarily screened in step 3, further remove the outliers with depth mutations. The screening method is to compare the 9-neighborhood around the target pixel. The screening criteria are: |d - d n | > trd, d is the depth estimation value of the target pixel, and d n is the depth estimation value of the nth pixel in the 9-neighborhood, where n takes values from 1 to 7, and the threshold trd is preferably 0.3m. When at least 7 pixels in the 9-neighborhood satisfy |d - d n | > trd, the target pixel is considered to be a mutated outlier and is deleted (as shown in FIG. 7); according to the above screening criteria, traverse horizontally and / or vertically in the pixel coordinate system at a preset step size to complete the preliminary screening of all pixels. Among them, the preset step size can be 1 pixel or 2 pixels, etc., and the present application does not limit this.

[0108] 5. Calculate the 3D coordinates (X, Y, Z) of the pixel using the normalized coordinates obtained in step 2. The calculation formula is: (X, Y, Z) = d(x, y, 1);

[0109] 6. Based on the 3D point cloud obtained from the 4 cameras in step 5, divide the point cloud space into voxels. The voxel density is determined according to the sampling density of the monocular point cloud (the empirical value is set to about 3-5 cm in length, width and height). For 3D points that fall within the same voxel unit, only the voxel center point is retained as the fusion result.

[0110] 7. After filtering and fusing in steps 3-5, obtain all valid pixels of the multi-view images. Based on all valid pixels of the multi-view images, obtain the point cloud distribution of objects around the vehicle in the current frame. Traverse the 3D points of the existing "bowl-shaped" model. For points where the distance between the 3D points of the "bowl-shaped" model and the 3D points in the monocular point cloud is less than a threshold (i.e., the two points fall within the same voxel unit), fuse them. The coordinates of the fused point are taken as the center coordinates of the voxel unit. Isolated points of the bowl-shaped model (points that have not been fused in multiple consecutive frames) are eliminated.

[0111] 8. Based on the fusion model obtained in step 7, perform texture mapping and calculate the pixel coordinates corresponding to each 3D point using the formula (u,v)=K(R(X,Y,Z)+t), where K is the intrinsic parameter matrix, and R and t are the extrinsic parameter rotation matrix and translation coefficient.

[0112] 9. Divide the common viewing area of ​​adjacent cameras into buffer zones. For 3D points within the buffer zone, repeat step 8 to calculate the pixel coordinates of the point on different panoramic original images, and then perform weighted superposition of pixel values. The weight is positively correlated with the size of the central angle of the 3D point projected onto the ground buffer zone (as shown in Figure 8).

[0113] The method for generating in-vehicle panoramic images in this application aims to improve the realism of in-vehicle surround-view 3D reconstruction and eliminate problems such as distortion and stretching in pseudo-3D schemes. Specifically:

[0114] 1. Existing 3D stitching methods suffer from significant stretching and deformation when dealing with tall objects (such as vehicles, walls, pillars, etc.) around the vehicle body. This application no longer uses a single fixed "bowl-shaped" model for texture binding. Instead, it estimates the three-dimensional spatial coordinates of pixels on the input panoramic image in real time to obtain the real object distribution. The shape of the "bowl-shaped" model is then adjusted to solve the problem of mismatch between the model and the object shape, thereby eliminating the stretching and deformation phenomenon.

[0115] 2. Most existing 3D stitching methods do not consider the real physical relationship between multiple cameras, resulting in obvious color differences at the stitching seams and significant discontinuities in 3D objects. This application no longer directly divides the area around the vehicle body according to the camera, but instead divides the overlapping visible area of ​​adjacent cameras into buffer zones (the size of the buffer zone is determined according to the camera distribution). For model points falling within the buffer zone, they are mapped to the textures on the two images simultaneously according to the distance. Finally, a weighted sum is calculated according to the distance, thereby improving the problems of obvious color differences and severe discontinuities at the stitching seams.

[0116] 3. To address the shortcomings of existing 3D stitching methods in failing to recover the true shape of objects, this application utilizes a monocular depth estimation network based on a fisheye camera to perform real-time depth estimation on four channels of fisheye images and generate monocular point clouds. By combining this network with certain strategies to filter out outliers with abnormal coordinates, and then fusing the monocular point clouds from multiple channels, the coordinates of the model points in the current "bowl-shaped model" are corrected, thereby achieving the goal of real-time fitting of the true shape of objects.

[0117] 4. To address the shortcomings of existing 3D stitching methods in failing to restore the true shape of objects, this application divides the four-way fisheye depth point cloud into three-dimensional spatial voxels according to a certain density, fuses and removes duplicates from the multi-way point cloud in the voxel space, and uses the fusion result to correct the current "bowl-shaped" model, thereby improving the realism of 3D scene reconstruction using the corrected "bowl-shaped" model.

[0118] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0119] In some embodiments, this application also provides an in-vehicle panoramic image generation device, comprising:

[0120] The data acquisition module is used to acquire multi-channel image data from multiple cameras in real time.

[0121] The point cloud data determination module is used to determine the point cloud data corresponding to the multi-channel image data;

[0122] The model correction module is used to correct the preset bowl-shaped model based on the point cloud data to obtain the corrected bowl-shaped model.

[0123] The texture mapping module is used to perform texture mapping on the corrected bowl-shaped model based on the multi-channel image data to obtain a dynamic real-world model.

[0124] The panoramic image generation module is used to generate in-vehicle panoramic images based on vehicle information and the dynamic real-scene model.

[0125] The vehicle-mounted panoramic image generation device provided in this application can be implemented as a vehicle-mounted 3D panoramic image stitching device that combines a monocular fisheye camera depth map. Based on the traditional stitching scheme, it can combine a monocular depth estimation network to generate monocular point clouds in real time. Then, it uses voxel division to fuse the point cloud results of multiple cameras. The fused point cloud results are used to guide the real-time dynamic adjustment of the preset "bowl-shaped" model, thereby fitting the real three-dimensional scene. Thus, it solves the problem of deformation and stretching caused by traditional 3D stitching and improves the realism of 3D scene reconstruction.

[0126] In some embodiments, determining the point cloud data corresponding to the plurality of road image data includes:

[0127] Filter the valid pixels in the multi-channel image data;

[0128] The point cloud data of objects around the vehicle body is determined based on the coordinate information of the effective pixels.

[0129] In some embodiments, filtering valid pixels from the multi-channel image data includes:

[0130] The depth estimate of pixels in the multi-channel image data is determined based on the depth estimation network;

[0131] Pixels whose depth estimates are within a preset threshold range are selected to determine the valid pixels.

[0132] In some embodiments, filtering valid pixels from the multi-channel image data includes:

[0133] The depth estimate of pixels in the multi-channel image data is determined based on the depth estimation network;

[0134] Filter pixels whose depth estimates are within a preset threshold range;

[0135] Delete pixels whose depth values ​​change abruptly from the pixels whose depth estimates are within a preset threshold range to determine the effective pixels.

[0136] In some embodiments, deleting pixels with abrupt changes in depth value among pixels whose depth estimates are within a preset threshold range includes:

[0137] The system iterates through pixels whose depth estimates fall within a preset threshold range within a preset neighborhood range to identify pixels with abrupt changes in depth values.

[0138] In some embodiments, filtering valid pixels from the multi-channel image data includes:

[0139] The depth estimate of pixels in the multi-channel image data is determined based on the depth estimation network;

[0140] Filter pixels whose depth estimates are within a preset threshold range;

[0141] Remove pixels whose depth estimates abruptly change within a preset threshold range;

[0142] For the point cloud spatial division of the pixels after deleting the mutated pixels, pixel fusion is performed to determine the effective pixels.

[0143] In some embodiments, for the point cloud spatial division voxels of pixels after deleting mutated pixels, pixel fusion is performed to determine the effective pixels, including:

[0144] Voxel partitioning of the point cloud for pixels after deleting mutated pixels;

[0145] For pixels falling within the same voxel unit, the center point of the same voxel unit is used as the fusion result to determine the effective pixels.

[0146] In some embodiments, correcting a preset 3D model based on the point cloud data to obtain a corrected 3D model includes:

[0147] The three-dimensional points of the preset three-dimensional model are traversed based on the point cloud data;

[0148] Pixels in the point cloud data that belong to the same preset range and 3D points on the preset 3D model are fused together.

[0149] In some embodiments, it also includes:

[0150] Three-dimensional points that are not fused after traversing the three-dimensional points of the preset three-dimensional model using multiple sets of point cloud data are deleted from the preset three-dimensional model; wherein, the multiple sets of point cloud data correspond to multiple sets of multi-channel image data.

[0151] In some embodiments, the multi-camera system includes adjacent first and second cameras, with an overlapping visible area between the first and second cameras, and the method further includes:

[0152] For overlapping pixels located within the overlapping visible area, obtain the first pixel value corresponding to the first camera and the second pixel value corresponding to the second camera;

[0153] The pixel value of the overlapping pixel is determined based on the first pixel value and the second pixel value.

[0154] In some embodiments, the method further includes:

[0155] Based on the distance of the overlapping pixels from the boundary of the overlapping visible area, a first weighted value corresponding to the first camera and a second weighted value corresponding to the majority of the second cameras are determined;

[0156] Determining the pixel value of the overlapping pixel based on the first pixel value and the second pixel value includes: performing a weighted summation on the first pixel value and the second pixel value using the first weighted value and the second weighted value to determine the pixel value of the overlapping pixel.

[0157] In some embodiments, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the vehicle panoramic image generation method described in any embodiment of this application.

[0158] In some embodiments, this application also provides a computer-readable storage medium storing a computer program / instructions thereon, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the vehicle panoramic image generation method described in any embodiment of this application.

[0159] In some embodiments, this application also provides a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the vehicle panoramic image generation method described in any embodiment of this application.

[0160] In some embodiments, this application also provides a mobile platform characterized by being equipped with the computer device described in any embodiment of the application. The mobile platform in this application includes, but is not limited to, vehicles, bipedal robots, quadrupedal robots, tracked robots, and aircraft.

[0161] The vehicle-mounted panoramic image generation device described in the above embodiments of this application can be used to execute the vehicle-mounted panoramic image generation of the embodiments of this application, and accordingly achieve the technical effects achieved by the vehicle-mounted panoramic image generation of the embodiments of this application, which will not be elaborated here. In the embodiments of this application, the relevant functional modules can be implemented by a hardware processor.

[0162] Figure 10 is a schematic diagram of the hardware structure of a computer device for executing a vehicle panoramic image generation method according to another embodiment of this application. As shown in Figure 10, the device includes:

[0163] One or more processors 1010 and memory 1020, with one processor 1010 as an example in Figure 10.

[0164] The device for performing the method of generating a panoramic image of a vehicle may further include an input device 1030 and an output device 1040.

[0165] The processor 1010, memory 1020, input device 1030 and output device 1040 can be connected by a bus or other means. Figure 10 shows an example of connection by a bus.

[0166] The memory 1020, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the vehicle panoramic image generation method in the embodiments of this application. The processor 1010 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1020, thereby implementing the vehicle panoramic image generation method of the above-described method embodiments.

[0167] The memory 1020 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the vehicle-mounted panoramic image generation device. Furthermore, the memory 1020 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1020 may optionally include memory remotely located relative to the processor 1010, and these remote memories can be connected to the vehicle-mounted panoramic image generation device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0168] The input device 1030 can receive input digital or character information, and generate signals related to user settings and function control of the in-vehicle panoramic image generation device. The output device 1040 may include a display device such as a display screen.

[0169] The one or more modules are stored in the memory 1020, and when executed by the one or more processors 1010, they execute the vehicle panoramic image generation method in any of the above method embodiments.

[0170] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0171] The computer device in this application embodiment exists in various forms, including but not limited to:

[0172] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0173] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0174] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0175] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0176] (5) Other electronic devices with data interaction functions.

[0177] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating a vehicle-mounted panoramic image, comprising: acquiring multi-channel image data collected by a plurality of cameras in real time; determining point cloud data corresponding to the multi-channel image data; correcting a preset three-dimensional model according to the point cloud data to obtain a corrected three-dimensional model; performing texture mapping on the corrected three-dimensional model according to the multi-channel image data to obtain a dynamic real scene model; generating a vehicle-mounted panoramic image according to vehicle information and the dynamic real scene model.

2. The method of claim 1, wherein, The determination of the point cloud data corresponding to the multi-channel image data comprises: screening valid pixels in the multi-channel image data; determining point cloud data of objects around a vehicle body according to coordinate information of the valid pixels.

3. The method of claim 2, wherein, The screening of the valid pixels in the multi-channel image data comprises: determining depth estimation values of pixels in the multi-channel image data according to a depth estimation network; screening pixels with depth estimation values within a preset threshold range to determine the valid pixels; or determining depth estimation values of pixels in the multi-channel image data according to a depth estimation network; screening pixels with depth estimation values within a preset threshold range; deleting pixels with depth values that mutate in the pixels with depth estimation values within the preset threshold range to determine the valid pixels; or determining depth estimation values of pixels in the multi-channel image data according to a depth estimation network; screening pixels with depth estimation values within a preset threshold range; deleting pixels with depth values that mutate in the pixels with depth estimation values within the preset threshold range; dividing voxels in a point cloud space for the pixels after the deletion of the pixels with mutated depth values, and performing pixel fusion to determine the valid pixels.

4. The method according to any one of claims 1 to 3, characterized in that, The correction of the preset three-dimensional model according to the point cloud data to obtain the corrected three-dimensional model comprises: traversing three-dimensional points of the preset three-dimensional model according to the point cloud data; fusing pixels in the point cloud data and the three-dimensional points on the preset three-dimensional model that belong to the same preset range.

5. The method of claim 4, wherein, Further comprising: deleting three-dimensional points that have not been fused from the preset three-dimensional model for the three-dimensional points of the preset three-dimensional model that are still not fused after being traversed by a plurality of groups of the point cloud data, wherein the plurality of groups of the point cloud data correspond to a plurality of groups of the multi-channel image data.

6. The method according to any one of claims 1-3, characterized in that, The plurality of cameras comprise adjacent first and second cameras, and the first and second cameras have an overlapping visible region, and the method further comprises: for overlapping pixels located in the overlapping visible region, acquiring a first pixel value corresponding to the first camera and a second pixel value corresponding to the second camera; determining a pixel value of the overlapping pixels according to the first and second pixel values. 7.A device for generating a vehicle-mounted panoramic image, comprising: a data acquisition module configured to acquire multi-channel image data collected by a plurality of cameras in real time; a point cloud data determination module configured to determine point cloud data corresponding to the multi-channel image data; a model correction module configured to correct a preset three-dimensional model according to the point cloud data to obtain a corrected three-dimensional model; a texture mapping module configured to perform texture mapping on the corrected three-dimensional model according to the multi-channel image data to obtain a dynamic real scene model; a panoramic image generation module configured to generate a vehicle-mounted panoramic image according to vehicle information and the dynamic real scene model.

8. The apparatus of claim 7, wherein, Determine the point cloud data corresponding to the multi-channel image data, comprising: Screening the valid pixels in the multi-channel image data; According to the coordinate information of the valid pixels, the point cloud data of the objects around the vehicle body is determined.

9. The apparatus of claim 8, wherein, Screening the valid pixels in the multi-channel image data, comprising: According to the depth estimation network, the depth estimation value of the pixels in the multi-channel image data is determined; the pixels with depth estimation values within the preset threshold range are screened to determine the valid pixels; or, According to the depth estimation network, the depth estimation value of the pixels in the multi-channel image data is determined; the pixels with depth estimation values within the preset threshold range are screened; the pixels with sudden changes in depth values in the pixels with depth estimation values within the preset threshold range are deleted to determine the valid pixels; or, According to the depth estimation network, the depth estimation value of the pixels in the multi-channel image data is determined; the pixels with depth estimation values within the preset threshold range are screened; the pixels with sudden changes in depth values in the pixels with depth estimation values within the preset threshold range are deleted; the point cloud space of the pixels after deleting the mutation pixels is divided into voxels, and the pixel fusion is performed to determine the valid pixels.

10. The apparatus of any one of claims 7-9, wherein, According to the point cloud data, the preset three-dimensional model is modified to obtain a modified three-dimensional model, comprising: According to the point cloud data, the three-dimensional points of the preset three-dimensional model are traversed; The pixels in the point cloud data and the three-dimensional points on the preset three-dimensional model within the same preset range are fused.

11. The apparatus of claim 10, wherein, Further comprising: For the three-dimensional points of the preset three-dimensional model that are still not fused after traversing the three-dimensional points of the preset three-dimensional model using multiple groups of point cloud data, the three-dimensional points are deleted from the preset three-dimensional model; wherein the multiple groups of point cloud data correspond to multiple groups of multi-channel image data.

12. The apparatus of any one of claims 7-9, wherein, The multi-channel camera includes adjacent first and second cameras, and the first and second cameras have an overlapping visible area, and the method further comprises: For the overlapping pixels located in the overlapping visible area, the first pixel value corresponding to the first camera and the second pixel value corresponding to the second camera are obtained; According to the first pixel value and the second pixel value, the pixel value of the overlapping pixel is determined.

13. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-12. The processor executes the computer program to realize the steps of the method of any one of claims 1-6.

14. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to realize the steps of the method of any one of claims 1-6.

15. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to realize the steps of the method of any one of claims 1-6.

16. A mobile platform, comprising: The computer equipment of claim 13 is installed. The computer equipment of claim 13 is installed.

Images