An image depth estimation method, device and equipment based on multi-modal fusion

Abstract

A kind of image depth estimation method, device and equipment based on multi-modal fusion, including the first common view image and the second common view image determined based on the common view area determined by first camera image and second camera image, the first common view image is subjected to depth estimation, and first depth map is obtained, the first common view image is subjected to semantic segmentation, and a plurality of first semantic regions are obtained, the second depth map and a plurality of second semantic regions corresponding to the second common view image are acquired, the first depth map is subjected to smoothing processing, and the first depth map after smoothing processing is obtained, the first depth map and the first depth map after smoothing processing are subjected to weighted fusion, and first fusion depth map is obtained.Second fusion depth map is generated based on the second depth map and a plurality of second semantic regions, and final depth estimation map is constructed according to first fusion depth map and second fusion depth map.The technical problem that accuracy and robustness are lower when single sensor is used to carry out depth estimation to the scene is solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an image depth estimation method, apparatus, and device based on multimodal fusion. Background Technology

[0002] In computer vision, depth estimation is a crucial task, aiming to extract 3D depth information of a scene from a 2D image. It plays a key role in numerous applications such as autonomous driving, robot navigation, and 3D reconstruction. Depth information helps robots understand their 3D environment, enabling them to perform navigation, obstacle avoidance, and object detection. Typically, depth estimation can be achieved using sensors such as monocular cameras, RGBD cameras, or LiDAR. RGBD cameras can directly obtain depth information from the corresponding image, but their measurement range is limited, and they are sensitive to lighting conditions outdoors. LiDAR is widely used for depth measurement in autonomous driving and industry, but it can only generate sparse 3D maps and cannot obtain detailed texture information of the scene. Monocular cameras are low-cost, small in size, and have a wide range of applications, so estimating dense depth maps from a single image is attracting increasing attention; however, depth estimation using monocular cameras still suffers from drawbacks such as low accuracy. Summary of the Invention

[0003] The main technical problem solved by this invention is the low accuracy and robustness of depth estimation of a scene when using a single sensor.

[0004] According to a first aspect, one embodiment provides an image depth estimation method based on multimodal fusion, applied in a heterogeneous camera system, the heterogeneous camera system including at least two different types of cameras; including:

[0005] The shared viewing area between the cameras is determined based on the pre-acquired first and second camera images; wherein, the first and second camera images of the target scene are generated by different types of cameras in the heterogeneous camera system, respectively.

[0006] Based on the shared viewing region, a first shared image and a second shared image are determined. Depth estimation is performed on the first shared image to obtain a first depth map. Semantic segmentation is performed on the first shared image to divide it into multiple first semantic regions, wherein each first semantic region has a corresponding semantic category label.

[0007] Depth estimation is performed on the second shared image to obtain a second depth map. Semantic segmentation is then performed on the second shared image to divide it into multiple second semantic regions, each of which has a corresponding semantic category label.

[0008] The first depth map is smoothed according to the semantic category label corresponding to each first semantic region to obtain a smoothed first depth map. The first depth map and the smoothed first depth map are then weighted and fused to obtain a first fused depth map.

[0009] A second fused depth map is generated based on the second depth map and the plurality of second semantic regions, and a final depth estimation map is constructed based on the first fused depth map and the second fused depth map.

[0010] In some embodiments, determining the shared viewing area between the cameras based on the pre-acquired first camera image and second camera image includes:

[0011] The first camera image and the second camera image are respectively subjected to distortion correction processing to obtain the first camera image and the second camera image after distortion correction.

[0012] The first camera image after distortion correction is projected onto the second camera image after distortion correction using a pre-calibrated extrinsic parameter matrix to obtain the first boundary coordinate point of the first camera image on the second camera image after distortion correction; wherein, the first boundary coordinate point includes a first left boundary coordinate point and a first right boundary coordinate point;

[0013] The distortion-corrected second camera image is projected onto the distortion-corrected first camera image using a pre-calibrated extrinsic parameter matrix to obtain the second boundary coordinate points of the distortion-corrected second camera image on the distortion-corrected first camera image; wherein, the second boundary coordinate points include a second left boundary coordinate point and a second right boundary coordinate point;

[0014] The intersection coordinate point is determined based on the first left boundary coordinate point, the first right boundary coordinate point, the second left boundary coordinate point, and the second right boundary coordinate point, and the area determined based on the intersection coordinate point is taken as the common viewing area between cameras.

[0015] In some embodiments, the heterogeneous camera system includes a fisheye camera and a pinhole camera, wherein the first camera image is acquired through the fisheye camera and the second camera image is acquired through the pinhole camera;

[0016] Determining the first shared image and the second shared image based on the shared viewing area includes:

[0017] The common viewing region is inversely transformed and then projected onto the first camera image to obtain the first common viewing image;

[0018] The shared viewing area is used as the second shared viewing image.

[0019] In some embodiments, the step of performing depth estimation on the first common image to obtain a first depth map includes:

[0020] The first common image is depth estimated using the trained first depth estimation model to obtain the first depth map;

[0021] The first depth estimation model is trained in the following manner:

[0022] Using the target image, the previous frame of the target image, and the next frame of the target image as training sample data, and using a loss function constructed based on the depth information corresponding to the training sample data, the transformation matrix corresponding to the training sample data, and the camera intrinsic parameter matrix as the training function, a preset depth estimation model is trained to obtain a trained first depth estimation model.

[0023] In some embodiments, smoothing the first depth map based on the semantic category label corresponding to each of the first semantic regions to obtain a smoothed first depth map includes:

[0024] Identify first semantic regions that have the same semantic category label among the plurality of first semantic regions;

[0025] The depth map regions in the first depth map that correspond to the first semantic region with the same semantic category label are taken as depth map regions belonging to the same target object.

[0026] The depth map regions belonging to the same target object in the first depth map are smoothed to obtain the smoothed first depth map.

[0027] In some embodiments, generating a second fused depth map based on the second depth map and the plurality of second semantic regions includes:

[0028] Identify second semantic regions within the plurality of second semantic regions that have the same semantic category label;

[0029] The depth map regions in the second depth map that correspond to the second semantic regions with the same semantic category labels are taken as depth map regions belonging to the same target object;

[0030] The depth map regions belonging to the same target object in the second depth map are smoothed to obtain a smoothed second depth map.

[0031] The second depth map and the smoothed second depth map are weighted and fused to obtain a second fused depth map.

[0032] In some embodiments, the weighted fusion of the first depth map and the smoothed first depth map to obtain a first fused depth map includes:

[0033] Determine the first weight corresponding to the first depth map and the second weight corresponding to the smoothed first depth map;

[0034] The first fused depth map is calculated based on the first weight, the second weight, the first depth map, and the smoothed first depth map.

[0035] According to a second aspect, one embodiment provides an image depth estimation device based on multimodal fusion, applied in a heterogeneous camera system, the heterogeneous camera system including at least two different types of cameras; including:

[0036] The common viewing area determination module is used to determine the common viewing area between cameras based on a pre-acquired first camera image and second camera image; wherein, the first camera image and the second camera image in the target scene are generated by different types of cameras in the heterogeneous camera system, respectively.

[0037] The data processing module is configured to determine a first shared image and a second shared image based on the shared viewing region; perform depth estimation on the first shared image to obtain a first depth map; perform semantic segmentation on the first shared image to divide it into multiple first semantic regions, wherein each first semantic region has a corresponding semantic category label; perform depth estimation on the second shared image to obtain a second depth map; and perform semantic segmentation on the second shared image to divide it into multiple second semantic regions, wherein each second semantic region has a corresponding semantic category label.

[0038] The image construction module is used to smooth the first depth map according to the semantic category label corresponding to each first semantic region to obtain a smoothed first depth map, perform weighted fusion of the first depth map and the smoothed first depth map to obtain a first fused depth map, generate a second fused depth map based on the second depth map and the plurality of second semantic regions, and construct a final depth estimation map based on the first fused depth map and the second fused depth map.

[0039] According to a third aspect, one embodiment provides an image depth estimation device based on multimodal fusion, comprising:

[0040] Memory, used to store programs;

[0041] A processor for implementing an image depth estimation method by executing a program stored in the memory.

[0042] According to a fourth aspect, one embodiment provides a computer-readable storage medium storing a program that can be executed by a processor to implement an image depth estimation method.

[0043] According to the multimodal fusion-based image depth estimation method, apparatus, device, and computer-readable storage medium of the above embodiments, since a first camera image and a second camera image of the target scene are generated by a heterogeneous camera system comprising at least two different types of cameras, subsequent depth estimation can refer to more comprehensive scene information combining the first and second camera images. The first depth map and the second depth map are smoothed based on multiple first semantic regions and multiple second semantic regions, respectively, to improve the accuracy and robustness of depth estimation. A final depth estimation map is constructed based on the first fused depth map and the second fused depth map, thereby improving the accuracy of depth estimation. Attached Figure Description

[0044] Figure 1 This is a flowchart of the image depth estimation method based on multimodal fusion according to an embodiment of this application;

[0045] Figure 2 This is a flowchart illustrating, in one embodiment, the determination of the shared viewing area between cameras based on a pre-acquired first camera image and a second camera image;

[0046] Figure 3 A flowchart of one embodiment of smoothing a first depth map based on the semantic category label corresponding to each first semantic region to obtain a smoothed first depth map;

[0047] Figure 4 This is a flowchart of a method for weighted fusion of a first depth map and a smoothed first depth map to obtain a first fused depth map in one embodiment.

[0048] Figure 5 This is a flowchart illustrating the generation of a second fused depth map based on a second depth map and multiple second semantic regions in one embodiment.

[0049] Figure 6 This is a schematic diagram of the structure of an image depth estimation device based on multimodal fusion according to an embodiment of this application. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0051] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0052] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0053] Please refer to Figure 1 In this embodiment of the invention, an image depth estimation method based on multimodal fusion is proposed and applied to a heterogeneous camera system, which includes at least two different types of cameras. The image depth estimation method includes steps S10 to S50, which are described in detail below.

[0054] Step S10: Determine the common viewing area between the cameras based on the pre-acquired first camera image and second camera image.

[0055] In some embodiments, a first camera image and a second camera image of the target scene are generated using different types of cameras in a heterogeneous camera system. For example, the heterogeneous camera system includes a pinhole camera and a fisheye camera. The first camera image is acquired using the fisheye camera, while the second camera image is acquired using the pinhole camera. The distance between the two cameras is small, and the time synchronization, intrinsic parameters, and extrinsic parameters between the cameras have been calibrated.

[0056] Please refer to Figure 2In some embodiments, step S10 determines the common viewing area between the cameras based on the pre-acquired first camera image and second camera image, including steps S11 to S14, which are described in detail below.

[0057] Step S11: Perform distortion correction processing on the first camera image and the second camera image respectively to obtain the first camera image and the second camera image after distortion correction.

[0058] Step S12: Project the first camera image after distortion correction onto the second camera image after distortion correction using the pre-calibrated extrinsic parameter matrix to obtain the first boundary coordinate point of the first camera image after distortion correction on the second camera image after distortion correction.

[0059] In some embodiments, the first boundary coordinate point includes a first left boundary coordinate point and a first right boundary coordinate point. For example, by projecting the distortion-corrected first camera image acquired by a fisheye camera onto the distortion-corrected second camera image acquired by a pinhole camera, the first left boundary coordinate point is obtained as (x... min_fish ,y min_fish The first right boundary coordinate point is (x max_fish ,y max_fish ).

[0060] Step S13: Project the distorted second camera image onto the distorted first camera image using the pre-calibrated extrinsic matrix to obtain the second boundary coordinate point of the distorted second camera image on the distorted first camera image.

[0061] In some embodiments, the second boundary coordinate point includes a second left boundary coordinate point and a second right boundary coordinate point. For example, by projecting the distortion-corrected second camera image acquired by the pinhole camera onto the distortion-corrected first camera image acquired by the fisheye camera, the second left boundary coordinate point is obtained as (x min_pin ,y min_pin The coordinates of the second right boundary point are (x max_pin ,y max_pin ).

[0062] Step S14: Determine the intersection coordinate point based on the first left boundary coordinate point, the first right boundary coordinate point, the second left boundary coordinate point, and the second right boundary coordinate point, and take the area determined by the intersection coordinate point as the common viewing area between cameras.

[0063] In some embodiments, the intersection coordinates are determined as follows:

[0064] [(max(x min_fish ,x min_pin ),max(y min_fish ,y min_pin )),(min(xmax_fish ,x max_pin ),

[0065] min(y max_fish ,y max_pin ))]

[0066] Among them, (x min_fish ,y min_fish (x) is the coordinate point of the first left boundary. max_fish ,y max_fish (x) is the coordinate point of the first right boundary. min_pin ,y min_pin (x) is the coordinate point of the second left boundary. max_pin ,y max_pin () represents the coordinates of the second right boundary point.

[0067] In some embodiments, the region determined by the intersection coordinates is used as the shared viewing area between the cameras. The shared viewing area refers to the portion that both cameras can see. Within this area, both cameras can see it; therefore, depth estimation within this area can eliminate irrelevant regions, reduce depth estimation errors caused by occlusion, and significantly reduce computational load while minimizing computational errors, thereby improving processing speed and efficiency. Furthermore, since the depth information within the shared viewing area comes from two different cameras, the depth estimates from both cameras can provide more reliable depth information.

[0068] Step S20: Determine the first shared image and the second shared image based on the shared viewing region, perform depth estimation on the first shared image to obtain the first depth map, and perform semantic segmentation on the first shared image to divide the first shared image into multiple first semantic regions, wherein each first semantic region has a corresponding semantic category label.

[0069] In some embodiments, the heterogeneous camera system includes a fisheye camera and a pinhole camera. The first camera image is acquired through the fisheye camera, and the second camera image is acquired through the pinhole camera. Therefore, the common viewing area is inversely transformed and the inversely transformed common viewing area is projected onto the first camera image to obtain the first common viewing image, and the common viewing area is used as the second common viewing image.

[0070] In some embodiments, depth estimation is performed on the first common image to obtain a first depth map, including:

[0071] The first depth map is obtained by using the trained first depth estimation model to estimate the depth of the first common image;

[0072] The first depth estimation model is trained in the following way:

[0073] Using the target image, the previous frame of the target image, and the next frame of the target image as training sample data, and using the loss function constructed based on the depth information, transformation matrix, and camera intrinsic parameter matrix corresponding to the training sample data as the training function, the preset depth estimation model is trained to obtain the first trained depth estimation model.

[0074] In some embodiments, it is assumed that the target image is T0, and the previous frame image of the target image T0 is T. -1 Let T1 be the next frame of the target image T0. Using the target image, its preceding frame, and its next frame as training sample data, the depth information corresponding to the training sample data is obtained through the target depth network and pose network. The transformation matrix corresponding to the training sample data is T. 0->1 or T 0->-1 .

[0075] In some embodiments, a loss function is constructed based on the depth information corresponding to the training sample data, the transformation matrix corresponding to the training sample data, and the camera intrinsic parameter matrix. The formula for the loss function is as follows:

[0076]

[0077] I t’->t =I t’ , <proj(D t ,T t->t, ,K)>

[0078] Among them, L p Denotes the loss function, D t T represents the depth information corresponding to the t-th frame image when it is used as training sample data. t->t’ Let I represent the transformation matrix, K represent the camera intrinsic parameter matrix, and I represent the transformation matrix. t’ I represents the reconstructed image of frame t'. t’->t This represents the image reconstruction process from frame t to frame t.

[0079] In some embodiments, the transformation matrix includes an axis-angle transformation matrix and a translation transformation matrix.

[0080] In some embodiments, since large-scale accurate depth estimation annotation information is unavailable, image reconstruction is used as a supervisory signal to train the depth estimation model. During training, when constructing the preset depth estimation model, image reconstruction is used as a supervisory signal. When using image reconstruction as a supervisory signal, the network structure of the preset depth estimation model often employs an encoding and decoding module. The encoding module uses a ResNet50 network structure, and the decoding module uses upsampled convolutional ResNet50 multi-scale feature maps for upsampling, so that the resolution of the image output by the decoding module is consistent with the resolution of the input image of the encoding module.

[0081] In some embodiments, the Omnidet algorithm is used to perform semantic segmentation on the first shared image, dividing it into multiple first semantic regions, each with a corresponding semantic category label. Semantic segmentation can provide detailed semantic information in the image, serving as a reference for the depth information of the same object in the subsequent smoothing process. It can also assign higher weights to semantic information with high deterministic depth information during the fusion process.

[0082] Step S30: Perform depth estimation on the second co-image to obtain a second depth map, and perform semantic segmentation on the second co-image to divide the second co-image into multiple second semantic regions, wherein each second semantic region has a corresponding semantic category label.

[0083] In some embodiments, a trained second depth estimation model is used to estimate the depth of the second common image to obtain a second depth map. The training process of the second depth estimation model is similar to that of the first depth estimation model, and will not be described again here.

[0084] In some embodiments, the segment anything algorithm is used to perform semantic segmentation on the second co-image to divide the second co-image into multiple second semantic regions, each of which has a corresponding semantic category label.

[0085] Step S40: Smooth the first depth map according to the semantic category label corresponding to each first semantic region to obtain the smoothed first depth map. Then, perform weighted fusion of the first depth map and the smoothed first depth map to obtain the first fused depth map.

[0086] Please refer to Figure 3 In some embodiments, step S40 smooths the first depth map according to the semantic category label corresponding to each first semantic region to obtain a smoothed first depth map, including steps S41 to S43, which are described in detail below.

[0087] Step S41: Identify first semantic regions with the same semantic category label among multiple first semantic regions.

[0088] Step S42: Take the depth map region in the first depth map that corresponds to the first semantic region with the same semantic category label as the depth map region belonging to the same target object.

[0089] Step S43: Smooth the depth map regions belonging to the same target object in the first depth map to obtain the smoothed first depth map.

[0090] In some embodiments, the same target object has the same semantic category label. During depth estimation, due to errors, the depth values ​​of the same target object may change abruptly, resulting in an unsmooth depth map. Therefore, depth map regions belonging to the same target object in the first depth map are determined based on multiple first semantic regions, and these regions are then smoothed to obtain a smoothed first depth map. The smoothing process is achieved through image median filtering.

[0091] Please refer to Figure 4 In some embodiments, step S40 performs weighted fusion of the first depth map and the smoothed first depth map to obtain a first fused depth map, including steps S44 to S45, which are described in detail below.

[0092] Step S44: Determine the first weight corresponding to the first depth map and the second weight corresponding to the smoothed first depth map.

[0093] Step S45: Calculate the first fused depth map based on the first weight, the second weight, the first depth map, and the smoothed first depth map.

[0094] In some embodiments, the first fusion depth map is calculated using the following formula:

[0095] depth_final=α*depth+(1-α)depth_blur

[0096] Where depth_final represents the first fused depth map, α represents the first weight, 1-α represents the second weight, depth represents the first depth map, and depth_blur represents the first depth map after smoothing.

[0097] In some embodiments, during the image depth estimation process, there may be regions with high certainty, such as the floor and walls in the image. In order to make the image depth estimation more accurate, regions with high certainty in the depth estimation can be given higher weights in the final fusion process to make them more certain.

[0098] Step S50: Generate a second fused depth map based on the second depth map and multiple second semantic regions, and construct a final depth estimation map based on the first fused depth map and the second fused depth map.

[0099] Please refer to Figure 5 In some embodiments, step S50 generates a second fused depth map based on the second depth map and multiple second semantic regions, including steps S51 to S54, which are described in detail below.

[0100] Step S51: Identify second semantic regions with the same semantic category label among multiple second semantic regions.

[0101] Step S52: Take the depth map region in the second depth map that corresponds to the second semantic region with the same semantic category label as the depth map region belonging to the same target object.

[0102] Step S53: Smooth the depth map regions belonging to the same target object in the second depth map to obtain the smoothed second depth map.

[0103] Step S54: Perform weighted fusion of the second depth map and the smoothed second depth map to obtain the second fused depth map.

[0104] In some embodiments, the specific process of generating the second fusion depth map is similar to that of generating the first fusion depth map, and will not be described again here.

[0105] In some embodiments, a final depth estimation map is constructed based on a first fused depth map and a second fused depth map, or the final depth estimation map can be constructed based on the average value of the first fused depth map and the second fused depth map.

[0106] In some embodiments, since most current depth estimation only applies to images generated by a single camera, there is limited research on depth estimation for heterogeneous camera systems composed of multiple cameras. These systems often suffer from differences in distortion parameters and field of view between cameras, leading to significant image discrepancies and hindering accurate depth estimation. While pinhole cameras exhibit less distortion, their limited field of view means that a single image captured by a pinhole camera contains limited information. Fisheye cameras suffer from significant radial distortion, and due to their distortion and wide-angle characteristics, they perform worse than pinhole cameras in high-precision scenarios. Therefore, this paper designs a heterogeneous camera system combining fisheye and pinhole cameras. The fisheye camera provides a wide-angle field of view, while the pinhole camera provides high-precision measurement. Combining the two provides more comprehensive scene information and is suitable for complex and diverse applications. By utilizing the shared field of view of the fisheye and pinhole cameras, and incorporating depth estimation and semantic segmentation, weighted fusion can be performed on targets with specific semantic categories, while simultaneously smoothing the depth information of the same target object in the image, thus improving the accuracy of depth estimation. Furthermore, in areas with sparse or repetitive textures, semantic segmentation information can be used to eliminate mismatches in the depth estimation process, and occluded areas can be better identified. Depth estimation can be performed based on contextual information, thereby improving the accuracy and robustness of depth estimation.

[0107] Please refer to Figure 6 This invention proposes an image depth estimation device based on multimodal fusion, which is applied in a heterogeneous camera system. The heterogeneous camera system includes at least two different types of cameras. The image depth estimation device includes a common viewing area determination module 10, a data processing module 20, and an image construction module 30, which are described in detail below.

[0108] The common viewing area determination module 10 is used to determine the common viewing area between cameras based on the pre-acquired first camera image and second camera image; wherein, the first camera image and second camera image of the target scene are generated by different types of cameras in the heterogeneous camera system respectively.

[0109] The data processing module 20 is used to determine a first shared image and a second shared image based on the shared viewing region, perform depth estimation on the first shared image to obtain a first depth map, perform semantic segmentation on the first shared image to divide the first shared image into multiple first semantic regions, wherein each first semantic region has a corresponding semantic category label; perform depth estimation on the second shared image to obtain a second depth map, and perform semantic segmentation on the second shared image to divide the second shared image into multiple second semantic regions, wherein each second semantic region has a corresponding semantic category label.

[0110] The image construction module 30 is used to smooth the first depth map according to the semantic category label corresponding to each first semantic region to obtain a smoothed first depth map, and to perform weighted fusion of the first depth map and the smoothed first depth map to obtain a first fused depth map; to generate a second fused depth map based on the second depth map and multiple second semantic regions, and to construct a final depth estimation map based on the first fused depth map and the second fused depth map.

[0111] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0112] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. An image depth estimation method based on multimodal fusion, applied in a heterogeneous camera system, wherein the heterogeneous camera system includes at least two cameras of different types; characterized in that, include: The shared viewing area between the cameras is determined based on the pre-acquired first and second camera images; wherein, the first and second camera images of the target scene are generated by different types of cameras in the heterogeneous camera system, respectively. Based on the shared viewing region, a first shared image and a second shared image are determined. Depth estimation is performed on the first shared image to obtain a first depth map. Semantic segmentation is performed on the first shared image to divide it into multiple first semantic regions, wherein each first semantic region has a corresponding semantic category label. Depth estimation is performed on the second shared image to obtain a second depth map. Semantic segmentation is then performed on the second shared image to divide it into multiple second semantic regions, each of which has a corresponding semantic category label. The first depth map is smoothed according to the semantic category label corresponding to each first semantic region to obtain a smoothed first depth map. The first depth map and the smoothed first depth map are then weighted and fused to obtain a first fused depth map. A second fused depth map is generated based on the second depth map and the plurality of second semantic regions, and a final depth estimation map is constructed based on the first fused depth map and the second fused depth map.

2. The image depth estimation method as described in claim 1, characterized in that, The step of determining the shared viewing area between the cameras based on the pre-acquired first camera image and second camera image includes: The first camera image and the second camera image are respectively subjected to distortion correction processing to obtain the first camera image and the second camera image after distortion correction. The first camera image after distortion correction is projected onto the second camera image after distortion correction using a pre-calibrated extrinsic parameter matrix to obtain the first boundary coordinate point of the first camera image on the second camera image after distortion correction; wherein, the first boundary coordinate point includes a first left boundary coordinate point and a first right boundary coordinate point; The distortion-corrected second camera image is projected onto the distortion-corrected first camera image using a pre-calibrated extrinsic parameter matrix to obtain the second boundary coordinate points of the distortion-corrected second camera image on the distortion-corrected first camera image; wherein, the second boundary coordinate points include a second left boundary coordinate point and a second right boundary coordinate point; The intersection coordinate point is determined based on the first left boundary coordinate point, the first right boundary coordinate point, the second left boundary coordinate point, and the second right boundary coordinate point, and the area determined based on the intersection coordinate point is taken as the common viewing area between cameras.

3. The image depth estimation method as described in claim 1, characterized in that, The heterogeneous camera system includes a fisheye camera and a pinhole camera. The first camera image is acquired through the fisheye camera, and the second camera image is acquired through the pinhole camera. Determining the first shared image and the second shared image based on the shared viewing area includes: The common viewing region is inversely transformed and then projected onto the first camera image to obtain the first common viewing image; The shared viewing area is used as the second shared viewing image.

4. The image depth estimation method as described in claim 1, characterized in that, The step of performing depth estimation on the first common image to obtain a first depth map includes: The first common image is depth estimated using the trained first depth estimation model to obtain the first depth map; The first depth estimation model is trained in the following manner: Using the target image, the previous frame of the target image, and the next frame of the target image as training sample data, and using a loss function constructed based on the depth information corresponding to the training sample data, the transformation matrix corresponding to the training sample data, and the camera intrinsic parameter matrix as the training function, a preset depth estimation model is trained to obtain a trained first depth estimation model.

5. The image depth estimation method as described in claim 1, characterized in that, The step of smoothing the first depth map according to the semantic category label corresponding to each first semantic region to obtain a smoothed first depth map includes: Identify first semantic regions that have the same semantic category label among the plurality of first semantic regions; The depth map regions in the first depth map that correspond to the first semantic region with the same semantic category label are taken as depth map regions belonging to the same target object. The depth map regions belonging to the same target object in the first depth map are smoothed to obtain the smoothed first depth map.

6. The image depth estimation method as described in claim 1, characterized in that, The generation of the second fused depth map based on the second depth map and the plurality of second semantic regions includes: Identify second semantic regions within the plurality of second semantic regions that have the same semantic category label; The depth map regions in the second depth map that correspond to the second semantic regions with the same semantic category labels are taken as depth map regions belonging to the same target object; The depth map regions belonging to the same target object in the second depth map are smoothed to obtain a smoothed second depth map. The second depth map and the smoothed second depth map are weighted and fused to obtain a second fused depth map.

7. The image depth estimation method as described in claim 1, characterized in that, The step of weightedly fusing the first depth map and the smoothed first depth map to obtain the first fused depth map includes: Determine the first weight corresponding to the first depth map and the second weight corresponding to the smoothed first depth map; The first fused depth map is calculated based on the first weight, the second weight, the first depth map, and the smoothed first depth map.

8. An image depth estimation device based on multimodal fusion, applied in a heterogeneous camera system, wherein the heterogeneous camera system includes at least two cameras of different types; characterized in that, include: The common viewing area determination module is used to determine the common viewing area between cameras based on a pre-acquired first camera image and second camera image; wherein, the first camera image and the second camera image in the target scene are generated by different types of cameras in the heterogeneous camera system, respectively. The data processing module is configured to determine a first shared image and a second shared image based on the shared viewing region; perform depth estimation on the first shared image to obtain a first depth map; perform semantic segmentation on the first shared image to divide it into multiple first semantic regions, wherein each first semantic region has a corresponding semantic category label; perform depth estimation on the second shared image to obtain a second depth map; and perform semantic segmentation on the second shared image to divide it into multiple second semantic regions, wherein each second semantic region has a corresponding semantic category label. The image construction module is used to smooth the first depth map according to the semantic category label corresponding to each first semantic region to obtain a smoothed first depth map, perform weighted fusion of the first depth map and the smoothed first depth map to obtain a first fused depth map, generate a second fused depth map based on the second depth map and the plurality of second semantic regions, and construct a final depth estimation map based on the first fused depth map and the second fused depth map.

9. An image depth estimation device based on multimodal fusion, characterized in that, include: Memory, used to store programs; A processor for implementing the image depth estimation method as described in any one of claims 1-7 by executing a program stored in the memory.

10. A computer-readable storage medium, characterized in that, The medium stores a program that can be executed by a processor to implement the image depth estimation method as described in any one of claims 1-7.

Images