Image depth estimation method, device and equipment based on multi-modal fusion

Abstract

The invention discloses an image depth estimation method, device and equipment based on multi-modal fusion, and the method comprises the steps: determining a first common-view image and a second common-view image based on a common-view region determined by a first camera image and a second camera image, carrying out the depth estimation of the first common-view image, obtaining a first depth image, carrying out the semantic segmentation of the first common-view image, and obtaining a second depth image; the method comprises the steps of obtaining a first depth map corresponding to a first common-view image and a plurality of semantic regions, obtaining a plurality of first semantic regions, obtaining a second depth map corresponding to a second common-view image and a plurality of second semantic regions, carrying out smoothing processing on the first depth map to obtain a first depth map after smoothing processing, and carrying out weighted fusion on the first depth map and the first depth map after smoothing processing to obtain a first fused depth map. And generating a second fusion depth map based on the second depth map and the plurality of second semantic regions, and constructing a final depth estimation map according to the first fusion depth map and the second fusion depth map. The technical problem that the accuracy and robustness are low when a single sensor is adopted to carry out depth estimation on a scene is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image depth estimation method, device and equipment based on multimodal fusion. Background Art

[0002] In computer vision, depth estimation is an important task that aims to extract the three-dimensional depth information of a scene from a two-dimensional image. It plays a key role in many applications such as autonomous driving, robot navigation, and 3D reconstruction. Depth information can help robots understand the three-dimensional environment and perform operations such as navigation, obstacle avoidance, and object detection. Usually, depth estimation can be achieved through sensors such as monocular cameras, RGBD cameras, or lidar. RGBD cameras can directly obtain the depth information of the corresponding image, but their measurement range is limited and they are sensitive to light outdoors; lidar is widely used for depth measurement in unmanned driving and industry, but it can only generate sparse three-dimensional maps and cannot obtain detailed texture information of the scene. Monocular cameras have low cost, small size, and wide application range, so estimating dense depth maps from a single image has attracted more and more attention. However, depth estimation using monocular cameras still has shortcomings such as low accuracy. Summary of the invention

[0003] The main technical problem solved by the present invention is that the accuracy and robustness of depth estimation of a scene using a single sensor are low.

[0004] According to a first aspect, an embodiment provides an image depth estimation method based on multimodal fusion, which is applied in a heterogeneous camera system, wherein the heterogeneous camera system includes at least two cameras of different types; including:

[0005] Determine a common viewing area between cameras based on a pre-acquired first camera image and a second camera image; wherein the first camera image and the second camera image under the target scene are generated by different types of cameras in the heterogeneous camera system respectively;

[0006] Determine a first co-viewing image and a second co-viewing image based on the co-viewing area, perform depth estimation on the first co-viewing image to obtain a first depth map, and perform semantic segmentation on the first co-viewing image to segment the first co-viewing image into a plurality of first semantic areas, wherein each of the first semantic areas has a corresponding semantic category label;

[0007] Performing depth estimation on the second co-viewing image to obtain a second depth map, and performing semantic segmentation on the second co-viewing image to segment the second co-viewing image into a plurality of second semantic regions, wherein each of the second semantic regions has a corresponding semantic category label;

[0008] Smoothing the first depth map according to the semantic category label corresponding to each of the first semantic regions to obtain a smoothed first depth map, and weighted fusion the first depth map and the smoothed first depth map to obtain a first fused depth map;

[0009] A second fused depth map is generated based on the second depth map and the plurality of second semantic regions, and a final depth estimation map is constructed according to the first fused depth map and the second fused depth map.

[0010] In some embodiments, determining the common viewing area between the cameras based on the pre-acquired first camera image and the second camera image includes:

[0011] Performing dedistortion processing on the first camera image and the second camera image respectively to obtain a dedistorted first camera image and a dedistorted second camera image;

[0012] Projecting the dedistorted first camera image onto the dedistorted second camera image using a pre-calibrated extrinsic matrix to obtain first boundary coordinate points of the dedistorted first camera image on the dedistorted second camera image; wherein the first boundary coordinate points include a first left boundary coordinate point and a first right boundary coordinate point;

[0013] Projecting the dedistorted second camera image onto the dedistorted first camera image using a pre-calibrated extrinsic matrix to obtain second boundary coordinate points of the dedistorted second camera image on the dedistorted first camera image; wherein the second boundary coordinate points include a second left boundary coordinate point and a second right boundary coordinate point;

[0014] An intersection coordinate point is determined according to the first left boundary coordinate point, the first right boundary coordinate point, the second left boundary coordinate point, and the second right boundary coordinate point, and an area determined according to the intersection coordinate point is used as a common viewing area between cameras.

[0015] In some embodiments, the heterogeneous camera system includes a fisheye camera and a pinhole camera, the first camera image is acquired by the fisheye camera, and the second camera image is acquired by the pinhole camera;

[0016] The determining of the first common viewing image and the second common viewing image based on the common viewing area includes:

[0017] Performing an inverse transformation on the common viewing area and projecting the inversely transformed common viewing area onto the first camera image to obtain a first common viewing image;

[0018] The common viewing area is used as a second common viewing image.

[0019] In some embodiments, performing depth estimation on the first co-viewed image to obtain a first depth map includes:

[0020] Performing depth estimation on the first co-viewing image using the trained first depth estimation model to obtain a first depth map;

[0021] The first depth estimation model is trained in the following manner:

[0022] A target image, a previous frame image of the target image and a subsequent frame image of the target image are used as training sample data, and a loss function constructed according to the depth information corresponding to the training sample data, the transformation matrix corresponding to the training sample data and the camera intrinsic parameter matrix is ​​used as a training function to train the preset depth estimation model to obtain a trained first depth estimation model.

[0023] In some embodiments, the step of smoothing the first depth map according to the semantic category label corresponding to each of the first semantic regions to obtain the smoothed first depth map includes:

[0024] Determine a first semantic region having the same semantic category label among the plurality of first semantic regions;

[0025] taking a depth map region in the first depth map corresponding to a first semantic region having the same semantic category label as a depth map region belonging to the same target object;

[0026] Smoothing is performed on depth map regions belonging to the same target object in the first depth map to obtain a smoothed first depth map.

[0027] In some embodiments, generating a second fused depth map based on the second depth map and the plurality of second semantic areas comprises:

[0028] determining a second semantic region having the same semantic category label among the plurality of second semantic regions;

[0029] taking a depth map region in the second depth map corresponding to a second semantic region having the same semantic category label as a depth map region belonging to the same target object;

[0030] Smoothing the depth map region belonging to the same target object in the second depth map to obtain a smoothed second depth map;

[0031] The second depth map and the smoothed second depth map are weightedly fused to obtain a second fused depth map.

[0032] In some embodiments, performing weighted fusion on the first depth map and the smoothed first depth map to obtain a first fused depth map includes:

[0033] Determine a first weight corresponding to the first depth map and a second weight corresponding to the smoothed first depth map;

[0034] A first fused depth map is calculated according to the first weight, the second weight, the first depth map, and the smoothed first depth map.

[0035] According to a second aspect, an embodiment provides an image depth estimation device based on multimodal fusion, which is applied in a heterogeneous camera system, wherein the heterogeneous camera system includes at least two cameras of different types; including:

[0036] A common viewing area determination module, configured to determine a common viewing area between cameras based on a pre-acquired first camera image and a second camera image; wherein the first camera image and the second camera image of a target scene are generated by different types of cameras in the heterogeneous camera system respectively;

[0037] A data processing module, configured to determine a first co-viewing image and a second co-viewing image based on the co-viewing area, perform depth estimation on the first co-viewing image to obtain a first depth map, perform semantic segmentation on the first co-viewing image to segment the first co-viewing image into a plurality of first semantic areas, wherein each of the first semantic areas has a corresponding semantic category label; perform depth estimation on the second co-viewing image to obtain a second depth map, perform semantic segmentation on the second co-viewing image to segment the second co-viewing image into a plurality of second semantic areas, wherein each of the second semantic areas has a corresponding semantic category label;

[0038] An image construction module is used to smooth the first depth map according to the semantic category label corresponding to each of the first semantic areas to obtain a smoothed first depth map, perform weighted fusion on the first depth map and the smoothed first depth map to obtain a first fused depth map; generate a second fused depth map based on the second depth map and the multiple second semantic areas, and construct a final depth estimation map based on the first fused depth map and the second fused depth map.

[0039] According to the third aspect, an embodiment provides an image depth estimation device based on multimodal fusion, including:

[0040] Memory, used to store programs;

[0041] The processor is used to implement the image depth estimation method by executing the program stored in the memory.

[0042] According to a fourth aspect, an embodiment provides a computer-readable storage medium, on which a program is stored, and the program can be executed by a processor to implement an image depth estimation method.

[0043] According to the method, device, apparatus and computer-readable storage medium for image depth estimation based on multimodal fusion of the above-mentioned embodiments, since the first camera image and the second camera image of the target scene are generated by a heterogeneous camera system constructed by at least two different types of cameras, the subsequent depth estimation can refer to the more comprehensive scene information combined with the first camera image and the second camera image. The first depth map and the second depth map are smoothed according to multiple first semantic areas and multiple second semantic areas respectively to improve the accuracy and robustness of the depth estimation, and the final depth estimation map is constructed according to the first fused depth map and the second fused depth map, thereby improving the accuracy of the depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of an image depth estimation method based on multimodal fusion according to an embodiment of the present application;

[0045] Figure 2 A flowchart of determining a common viewing area between cameras according to a pre-acquired first camera image and a second camera image in an embodiment;

[0046] Figure 3 A flowchart of performing smoothing on the first depth map according to the semantic category label corresponding to each first semantic area in an embodiment to obtain the smoothed first depth map;

[0047] Figure 4 A flowchart of performing weighted fusion on the first depth map and the smoothed first depth map to obtain a first fused depth map in an embodiment;

[0048] Figure 5 A flowchart of generating a second fused depth map based on a second depth map and a plurality of second semantic regions in an embodiment;

[0049] Figure 6 Schematic diagram of the structure of an image depth estimation device based on multimodal fusion according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0051] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0052] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0053] Please refer to Figure 1 In an embodiment of the present invention, a method for image depth estimation based on multimodal fusion is proposed, which is applied to a heterogeneous camera system. The heterogeneous camera system includes at least two cameras of different types. The method for image depth estimation includes steps S10 to S50, which are described in detail below.

[0054] Step S10: determining a common viewing area between cameras according to the pre-acquired first camera image and the second camera image.

[0055] In some embodiments, a first camera image and a second camera image of a target scene are generated by different types of cameras in a heterogeneous camera system. For example, the heterogeneous camera system includes a pinhole camera and a fisheye camera, the first camera image is obtained by the fisheye camera, and the second camera image is obtained by the pinhole camera, wherein the distance between the two cameras is small, and the time synchronization between the cameras, the intrinsic parameters, and the extrinsic parameters between the cameras have been calibrated.

[0056] Please refer to Figure 2In some embodiments, step S10 determines the common viewing area between cameras based on the pre-acquired first camera image and the second camera image, including steps S11 to S14, which are described in detail below.

[0057] Step S11: performing dedistortion processing on the first camera image and the second camera image respectively to obtain a dedistorted first camera image and a dedistorted second camera image.

[0058] Step S12: Projecting the dedistorted first camera image onto the dedistorted second camera image using a pre-calibrated extrinsic matrix to obtain first boundary coordinate points of the dedistorted first camera image on the dedistorted second camera image.

[0059] In some embodiments, the first boundary coordinate point includes a first left boundary coordinate point and a first right boundary coordinate point. For example, the first camera image obtained by the fisheye camera after dedistortion is projected onto the second camera image obtained by the pinhole camera after dedistortion, and the first left boundary coordinate point is obtained as (x min_fish ,y min_fish ), the first right boundary coordinate point is (x max_fish ,y max_fish ).

[0060] Step S13: projecting the dedistorted second camera image onto the dedistorted first camera image using a pre-calibrated extrinsic matrix to obtain second boundary coordinate points of the dedistorted second camera image on the dedistorted first camera image.

[0061] In some embodiments, the second boundary coordinate point includes a second left boundary coordinate point and a second right boundary coordinate point. For example, the second camera image obtained by the pinhole camera after dedistortion is projected onto the first camera image obtained by the fisheye camera after dedistortion, and the second left boundary coordinate point is obtained as (x min_pin ,y min_pin ), the second right boundary coordinate point is (x max_pin ,y max_pin ).

[0062] Step S14: determining an intersection coordinate point according to the first left boundary coordinate point, the first right boundary coordinate point, the second left boundary coordinate point, and the second right boundary coordinate point, and using the area determined according to the intersection coordinate point as the common view area between the cameras.

[0063] In some embodiments, the intersection coordinate point is determined by:

[0064] [(max(x min_fish ,x min_pin ),max(y min_fish ,y min_pin )),(min(xmax_fish ,x max_pin ),

[0065] min(y max_fish ,y max_pin ))]

[0066] Among them, (x min_fish ,y min_fish ) is the first left boundary coordinate point, (x max_fish ,y max_fish ) is the first right boundary coordinate point, (x min_pin ,y min_pin ) is the second left boundary coordinate point, (x max_pin ,y max_pin ) is the second right boundary coordinate point.

[0067] In some embodiments, the area determined according to the intersection coordinate points is used as the common view area between the cameras, and the common view area refers to the part that can be seen by both cameras. In the common view area, both cameras can see the area, so depth estimation in the area can eliminate irrelevant areas, reduce depth estimation errors caused by occlusion, and significantly reduce the amount of calculation while reducing calculation errors, thereby improving processing speed and efficiency. At the same time, since the depth information in the common view area comes from two different cameras, the depth estimation of the two cameras can provide more reliable depth information.

[0068] Step S20: Determine the first co-viewing image and the second co-viewing image based on the co-viewing area, perform depth estimation on the first co-viewing image to obtain a first depth map, and perform semantic segmentation on the first co-viewing image to segment the first co-viewing image into multiple first semantic areas, each of which has a corresponding semantic category label.

[0069] In some embodiments, a heterogeneous camera system includes a fisheye camera and a pinhole camera, wherein the first camera image is acquired through the fisheye camera and the second camera image is acquired through the pinhole camera. Therefore, the common view area is inversely transformed and the inversely transformed common view area is projected onto the first camera image to obtain a first common view image, and the common view area is used as the second common view image.

[0070] In some embodiments, performing depth estimation on the first co-viewing image to obtain a first depth map includes:

[0071] Performing depth estimation on the first co-viewing image using the trained first depth estimation model to obtain a first depth map;

[0072] Among them, the first depth estimation model is trained in the following way:

[0073] The target image, the previous frame image of the target image and the next frame image of the target image are used as training sample data, and the loss function constructed according to the depth information corresponding to the training sample data, the transformation matrix corresponding to the training sample data and the camera intrinsic parameter matrix is ​​used as the training function to train the preset depth estimation model to obtain a trained first depth estimation model.

[0074] In some embodiments, it is assumed that the target image is T0 and the previous frame image of the target image T0 is T -1 , the next frame image of the target image T0 is T1, and the target image, the previous frame image of the target image and the next frame image of the target image are used as training sample data. Then, according to the target depth network and the pose network, the depth information corresponding to the training sample data is obtained as follows: The transformation matrix corresponding to the training sample data is T 0->1 or T 0->-1 .

[0075] In some embodiments, a loss function is constructed based on the depth information corresponding to the training sample data, the transformation matrix corresponding to the training sample data, and the camera intrinsic parameter matrix, wherein the formula of the loss function is as follows:

[0076]

[0077] I t’->t =I t’ , <proj(D t ,T t->t, ,K)>

[0078] Among them, L p represents the loss function, D t Indicates the depth information corresponding to the t-th frame image as training sample data, T t->t’ represents the transformation matrix, K represents the camera intrinsic parameter matrix, I t’ represents the reconstructed image of the t'th frame, I t’->t Represents the image reconstruction process from the t,th frame to the t,th frame.

[0079] In some embodiments, the transformation matrix includes an axis-angle change matrix and a translation change matrix.

[0080] In some embodiments, since it is impossible to obtain large-scale accurate depth estimation annotation information, image reconstruction is used as a supervisory signal to train the depth estimation model. During the training process, when constructing a preset depth estimation model, image reconstruction is used as a supervisory signal. When image reconstruction is used as a supervisory signal, the network structure of the preset depth estimation model often adopts a codec module, wherein the encoding module adopts a Resnet50 network structure, and the decoding module adopts an upsampled convolution Resnet50 multi-scale feature map for upsampling, so that the resolution of the image output by the decoding module is consistent with the resolution of the input image of the encoding module.

[0081] In some embodiments, the omnidet algorithm is used to perform semantic segmentation on the first co-viewing image to segment the first co-viewing image into a plurality of first semantic regions, wherein each first semantic region has a corresponding semantic category label. Semantic segmentation can provide detailed semantic information in the image as a reference for depth information of the same object in a subsequent smoothed image, and can also give a higher weight to semantic information with high certainty of depth information during the fusion process.

[0082] Step S30: performing depth estimation on the second co-viewing image to obtain a second depth map, and performing semantic segmentation on the second co-viewing image to segment the second co-viewing image into a plurality of second semantic regions, wherein each second semantic region has a corresponding semantic category label.

[0083] In some embodiments, a trained second depth estimation model is used to perform depth estimation on the second co-viewing image to obtain a second depth map, wherein the training process of the second depth estimation model is similar to the training process of the first depth estimation model and will not be repeated here.

[0084] In some embodiments, the second co-viewing image is semantically segmented using a segment anything algorithm to segment the second co-viewing image into a plurality of second semantic regions, wherein each second semantic region has a corresponding semantic category label.

[0085] Step S40: smoothing the first depth map according to the semantic category label corresponding to each first semantic area to obtain a smoothed first depth map, and weighted fusion of the first depth map and the smoothed first depth map to obtain a first fused depth map.

[0086] Please refer to Figure 3 In some embodiments, step S40 performs smoothing on the first depth map according to the semantic category label corresponding to each first semantic area to obtain a smoothed first depth map, including steps S41 to S43, which are described in detail below.

[0087] Step S41: determining first semantic regions having the same semantic category label among a plurality of first semantic regions.

[0088] Step S42: taking the depth map region corresponding to the first semantic region having the same semantic category label in the first depth map as the depth map region belonging to the same target object.

[0089] Step S43: performing smoothing on the depth map regions belonging to the same target object in the first depth map to obtain a smoothed first depth map.

[0090] In some embodiments, the same target object has the same semantic category label. During the depth estimation process, due to the existence of errors, the depth value in the same target object may suddenly change, resulting in an unsmooth depth map. Therefore, the depth map area belonging to the same target object in the first depth map is determined according to the multiple first semantic areas, and the depth map area belonging to the same target object in the first depth map is smoothed to obtain a smoothed first depth map. The smoothing is achieved by image median filtering.

[0091] Please refer to Figure 4 In some embodiments, step S40 performs weighted fusion on the first depth map and the smoothed first depth map to obtain a first fused depth map, including steps S44 to S45, which are described in detail below.

[0092] Step S44: determining a first weight corresponding to the first depth map and a second weight corresponding to the smoothed first depth map.

[0093] Step S45: Calculate a first fused depth map according to the first weight, the second weight, the first depth map, and the smoothed first depth map.

[0094] In some embodiments, the first fused depth map is calculated by the following formula:

[0095] depth_final=α*depth+(1-α)depth_blur

[0096] Among them, depth_final represents the first fused depth map, α represents the first weight, 1-α represents the second weight, depth represents the first depth map, and depth_blur represents the first depth map after smoothing.

[0097] In some embodiments, during the image depth estimation process, there will be areas with higher certainty, such as the floor and walls in the image. In order to make the image depth estimation more accurate, the areas with higher certainty in the depth estimation can be given a higher weight in the final fusion process to make their certainty higher.

[0098] Step S50: generating a second fused depth map based on the second depth map and the plurality of second semantic regions, and constructing a final depth estimation map according to the first fused depth map and the second fused depth map.

[0099] Please refer to Figure 5 In some embodiments, step S50 generates a second fused depth map based on the second depth map and the plurality of second semantic regions, including steps S51 to S54, which are described in detail below.

[0100] Step S51: determining second semantic regions having the same semantic category label among a plurality of second semantic regions.

[0101] Step S52: taking the depth map region in the second depth map corresponding to the second semantic region having the same semantic category label as the depth map region belonging to the same target object.

[0102] Step S53: Smoothing the depth map regions belonging to the same target object in the second depth map to obtain a smoothed second depth map.

[0103] Step S54: performing weighted fusion on the second depth map and the smoothed second depth map to obtain a second fused depth map.

[0104] In some embodiments, the specific process of generating the second fused depth map is similar to the process of generating the first fused depth map, which will not be described in detail here.

[0105] In some embodiments, a final depth estimation map is constructed according to the first fused depth map and the second fused depth map, and the final depth estimation map may be constructed according to an average value of the first fused depth map and the second fused depth map.

[0106] In some embodiments, since most current depth estimation is only for images generated by a single camera, there are few studies on depth estimation of heterogeneous camera systems composed of multiple cameras. The two usually have different distortion parameters and different field of view angles between cameras, resulting in large imaging differences and inability to perform accurate depth estimation. Although the pinhole camera has less distortion, its field of view is small. The information contained in a single image taken by the pinhole camera is limited. The image taken by the fisheye camera has significant radial distortion. Due to the characteristics of distortion and wide angle, the fisheye camera does not perform as well as the pinhole camera in scenes with high precision requirements. Therefore, a heterogeneous camera system combining a fisheye camera and a pinhole camera is designed. The fisheye camera provides a wide-angle field of view and the pinhole camera provides high-precision measurement. The combination of the two can obtain more comprehensive scene information and is suitable for complex and diverse application scenarios. Through the common viewing area of ​​the fisheye camera and the pinhole camera, combined with the idea of ​​depth estimation and semantic segmentation, weighted fusion can be performed for targets of special semantic categories, and the depth information of the same target in the image can be smoothed, so that the accuracy of depth estimation is improved. In areas with sparse texture or repeated texture, the semantic segmentation information can be used to eliminate mismatches in the depth estimation process, and the occluded areas can be better identified. Depth estimation is performed based on contextual information, thereby improving the accuracy and robustness of depth estimation.

[0107] Please refer to Figure 6 An embodiment of the present invention proposes an image depth estimation device based on multimodal fusion, which is applied in a heterogeneous camera system, wherein the heterogeneous camera system includes at least two cameras of different types; the image depth estimation device includes a common view area determination module 10, a data processing module 20 and an image construction module 30, which are described in detail below.

[0108] The common viewing area determination module 10 is used to determine the common viewing area between cameras based on the pre-acquired first camera image and the second camera image; wherein the first camera image and the second camera image under the target scene are generated by different types of cameras in the heterogeneous camera system respectively.

[0109] The data processing module 20 is used to determine the first common view image and the second common view image based on the common view area, perform depth estimation on the first common view image to obtain a first depth map, perform semantic segmentation on the first common view image to segment the first common view image into multiple first semantic areas, wherein each first semantic area has a corresponding semantic category label; perform depth estimation on the second common view image to obtain a second depth map, perform semantic segmentation on the second common view image to segment the second common view image into multiple second semantic areas, wherein each second semantic area has a corresponding semantic category label.

[0110] The image construction module 30 is used to smooth the first depth map according to the semantic category label corresponding to each first semantic area to obtain the smoothed first depth map, perform weighted fusion on the first depth map and the smoothed first depth map to obtain a first fused depth map; generate a second fused depth map based on the second depth map and multiple second semantic areas, and construct a final depth estimation map based on the first fused depth map and the second fused depth map.

[0111] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0112] The above specific examples are used to illustrate the present invention, which are only used to help understand the present invention and are not used to limit the present invention. For those skilled in the art to which the present invention belongs, some simple deductions, modifications or substitutions can be made according to the idea of ​​the present invention.

Claims

1. An image depth estimation method based on multimodal fusion, applied in a heterogeneous camera system, wherein the heterogeneous camera system includes at least two cameras of different types; characterized in that: include: Determine a common viewing area between cameras based on a pre-acquired first camera image and a second camera image; wherein the first camera image and the second camera image under the target scene are generated by different types of cameras in the heterogeneous camera system respectively; Determine a first co-viewing image and a second co-viewing image based on the co-viewing area, perform depth estimation on the first co-viewing image to obtain a first depth map, and perform semantic segmentation on the first co-viewing image to segment the first co-viewing image into a plurality of first semantic areas, wherein each of the first semantic areas has a corresponding semantic category label; Performing depth estimation on the second co-viewing image to obtain a second depth map, and performing semantic segmentation on the second co-viewing image to segment the second co-viewing image into a plurality of second semantic regions, wherein each of the second semantic regions has a corresponding semantic category label; Smoothing the first depth map according to the semantic category label corresponding to each of the first semantic regions to obtain a smoothed first depth map, and weighted fusion the first depth map and the smoothed first depth map to obtain a first fused depth map; A second fused depth map is generated based on the second depth map and the plurality of second semantic regions, and a final depth estimation map is constructed according to the first fused depth map and the second fused depth map.

2. The image depth estimation method according to claim 1, characterized in that: The determining of the common viewing area between the cameras according to the pre-acquired first camera image and the second camera image comprises: Performing dedistortion processing on the first camera image and the second camera image respectively to obtain a dedistorted first camera image and a dedistorted second camera image; Projecting the dedistorted first camera image onto the dedistorted second camera image using a pre-calibrated extrinsic matrix to obtain first boundary coordinate points of the dedistorted first camera image on the dedistorted second camera image; wherein the first boundary coordinate points include a first left boundary coordinate point and a first right boundary coordinate point; Projecting the dedistorted second camera image onto the dedistorted first camera image using a pre-calibrated extrinsic matrix to obtain second boundary coordinate points of the dedistorted second camera image on the dedistorted first camera image; wherein the second boundary coordinate points include a second left boundary coordinate point and a second right boundary coordinate point; An intersection coordinate point is determined according to the first left boundary coordinate point, the first right boundary coordinate point, the second left boundary coordinate point, and the second right boundary coordinate point, and an area determined according to the intersection coordinate point is used as a common viewing area between cameras.

3. The image depth estimation method according to claim 1, characterized in that: The heterogeneous camera system includes a fisheye camera and a pinhole camera, the first camera image is obtained by the fisheye camera, and the second camera image is obtained by the pinhole camera; The determining of the first common viewing image and the second common viewing image based on the common viewing area includes: Performing an inverse transformation on the common viewing area and projecting the inversely transformed common viewing area onto the first camera image to obtain a first common viewing image; The common viewing area is used as a second common viewing image.

4. The image depth estimation method according to claim 1, characterized in that: The performing depth estimation on the first co-viewed image to obtain a first depth map includes: Performing depth estimation on the first co-viewing image using the trained first depth estimation model to obtain a first depth map; The first depth estimation model is trained in the following manner: A target image, a previous frame image of the target image and a subsequent frame image of the target image are used as training sample data, and a loss function constructed according to the depth information corresponding to the training sample data, the transformation matrix corresponding to the training sample data and the camera intrinsic parameter matrix is ​​used as a training function to train the preset depth estimation model to obtain a trained first depth estimation model.

5. The image depth estimation method according to claim 1, characterized in that: The step of smoothing the first depth map according to the semantic category label corresponding to each of the first semantic regions to obtain the smoothed first depth map includes: Determine a first semantic region having the same semantic category label among the plurality of first semantic regions; taking a depth map region in the first depth map corresponding to a first semantic region having the same semantic category label as a depth map region belonging to the same target object; Smoothing is performed on depth map regions belonging to the same target object in the first depth map to obtain a smoothed first depth map.

6. The image depth estimation method according to claim 1, characterized in that: The generating a second fused depth map based on the second depth map and the plurality of second semantic areas comprises: determining a second semantic region having the same semantic category label among the plurality of second semantic regions; taking a depth map region in the second depth map corresponding to a second semantic region having the same semantic category label as a depth map region belonging to the same target object; Smoothing the depth map region belonging to the same target object in the second depth map to obtain a smoothed second depth map; The second depth map and the smoothed second depth map are weightedly fused to obtain a second fused depth map.

7. The image depth estimation method according to claim 1, characterized in that: The step of performing weighted fusion on the first depth map and the smoothed first depth map to obtain a first fused depth map includes: Determine a first weight corresponding to the first depth map and a second weight corresponding to the smoothed first depth map; A first fused depth map is calculated according to the first weight, the second weight, the first depth map, and the smoothed first depth map.

8. An image depth estimation device based on multimodal fusion, applied in a heterogeneous camera system, wherein the heterogeneous camera system includes at least two cameras of different types; characterized in that: include: A common viewing area determination module, configured to determine a common viewing area between cameras based on a pre-acquired first camera image and a second camera image; wherein the first camera image and the second camera image of a target scene are generated by different types of cameras in the heterogeneous camera system respectively; A data processing module, configured to determine a first co-viewing image and a second co-viewing image based on the co-viewing area, perform depth estimation on the first co-viewing image to obtain a first depth map, perform semantic segmentation on the first co-viewing image to segment the first co-viewing image into a plurality of first semantic areas, wherein each of the first semantic areas has a corresponding semantic category label; perform depth estimation on the second co-viewing image to obtain a second depth map, perform semantic segmentation on the second co-viewing image to segment the second co-viewing image into a plurality of second semantic areas, wherein each of the second semantic areas has a corresponding semantic category label; An image construction module is used to smooth the first depth map according to the semantic category label corresponding to each of the first semantic areas to obtain a smoothed first depth map, perform weighted fusion on the first depth map and the smoothed first depth map to obtain a first fused depth map; generate a second fused depth map based on the second depth map and the multiple second semantic areas, and construct a final depth estimation map based on the first fused depth map and the second fused depth map.

9. An image depth estimation device based on multimodal fusion, characterized in that: include: Memory, used to store programs; A processor, configured to implement the image depth estimation method according to any one of claims 1 to 7 by executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The medium stores a program, which can be executed by a processor to implement the image depth estimation method according to any one of claims 1 to 7.

Images