A point cloud and image dynamic depth map alignment method and device

Abstract

The application discloses a point cloud and image dynamic depth map alignment method and device. The method comprises the following steps: S1, collecting point cloud data of one view angle of a target, images of multiple view angles and corresponding camera parameters of the images; S2, inputting the point cloud data, the images and the camera parameters into a depth alignment model to obtain a 3D target detection vector. By utilizing the cooperative advantages of cross-view depth map estimation, the depth map alignment of each view angle of the image and the point cloud data is sequentially selected, so that the accuracy of the enhanced depth map alignment estimation is realized, and the projection distortion problem caused by the sparsity of the point cloud is solved. Meanwhile, a dynamic gating network is constructed to determine which branch is selected. The gating network can activate part of the branches in all image features according to a probability vector, and the branches that are not activated will not participate in the operation, so that the calculation amount is reduced, the operation efficiency is improved, and the problem of excessive calculation amount of multiple alignment branches corresponding to multiple view angles of the images is solved.

Description

Technical Field

[0001] This invention relates to the alignment of point clouds with dynamic depth maps of images, and particularly to a method and apparatus for aligning point clouds with dynamic depth maps of images. Background Technology

[0002] With the rapid development of drone technology, the phenomenon of drones flying illegally has become frequent. Accurate 3D target detection for drones has become a core requirement for drone target detection and identification. The key challenge of this task lies in how to accurately acquire the target's three-dimensional spatial information (including position, size, and orientation) and maintain robustness in complex environments. Traditional single-sensor solutions often struggle to balance accuracy and reliability, while multimodal sensor fusion technology offers a new approach to solving this problem.

[0003] Modern UAV detection systems typically employ a multi-sensor collaborative approach, with cameras and radar (including lidar and millimeter-wave radar) being the two primary perception modalities. Multi-view images captured by cameras from multiple perspectives provide rich texture and color information, enabling the identification of surface features and semantic information of the target. Meanwhile, point cloud data acquired by radar provides precise geometric structure and spatial information. These two modalities are complementary in information representation, and their combination can significantly enhance the performance ceiling of the detection system.

[0004] In existing technologies, point cloud data is typically projected as a Bird's-Eye View (BEV) or a Distance View (RV) to reduce computational complexity. Multi-view fusion methods utilize both BEV and RV perspectives for detection, and combine Dempster-Shafer evidence theory to improve detection accuracy in occluded scenes. BEVFusion addresses the projection distortion problem caused by projecting point cloud data into a 2D image space. Its proposed method maps LiDAR and camera features to a unified BEV (Bird's-Eye View) space with less projection distortion, while employing an independent dual-branch architecture for modal fusion. LXL uses 4D imaging radar and camera fusion, proposing a "radar-occupied auxiliary depth sampling" strategy to optimize viewpoint transitions. MSHP3D, through a multi-stage cross-modal fusion module, combines image semantic enhancement of point cloud features, demonstrating excellent performance in indoor detection tasks. Lift-Attend-Splat abandons monocular depth estimation and instead uses a Transformer to directly fuse BEV features, significantly improving small target detection performance.

[0005] However, during BEV projection, fine-grained features in the vertical direction (Z-axis) are highly compressed, resulting in the loss of vertical information of the UAV target. In wide areas, UAV point cloud information is relatively sparse, and projection onto it causes projection distortion, making it difficult to accurately align with multi-view images. Summary of the Invention

[0006] To address the aforementioned issues, this application provides a method and apparatus for aligning point clouds with dynamic depth maps of images.

[0007] This invention provides the following technical solution:

[0008] A method for aligning 3D point clouds with dynamic depth maps of images includes the following steps:

[0009] S1. Collect point cloud data of the target from one viewpoint, images from multiple viewpoints, and camera parameters corresponding to each image.

[0010] S2. Input the point cloud data, image, and camera parameters into the depth alignment model to obtain the 3D object detection vector;

[0011] The depth alignment model selects branch images using a gated network based on gated image features, obtains reference and source images based on the branch images, obtains a predicted depth map based on the reference and source images, obtains a true depth map based on the camera parameters and point cloud data corresponding to the reference image, and obtains a 3D target detection vector based on the image feature set of the gated image features, the true depth map, and the branch images.

[0012] Furthermore, the depth alignment model is constructed through the following steps:

[0013] (1) Construct the format of the input dataset and the format of the output dataset;

[0014] (2) Construct a first convolutional layer for feature processing of images from multiple perspectives to obtain gated image features of images from multiple perspectives;

[0015] (3) Construct a gating network to convert the gating image features into probability vectors, and then find the images corresponding to the highest probability values ​​based on the probability vectors, i.e., branch images;

[0016] (4) Construct a second convolutional layer to obtain image feature sets for each image in the branch images, divide the branch images into reference images and source images, and obtain the predicted depth map through the stereo reconstruction network; obtain the real depth map based on the camera parameters corresponding to the reference image and the point cloud data;

[0017] (5) Tensor concatenation of the image feature sets of the branch images is performed to obtain multi-branch image features; tensor concatenation of the gated image features, the true depth map and the multi-branch image features is performed to obtain fused features; the fused features are input into the second convolutional layer to obtain output features; the input is input into the global average pooling layer to obtain 3D object detection vector;

[0018] (6) Input the training dataset and the test dataset into the depth alignment model;

[0019] (7) Calculate the joint loss function based on the predicted depth map and the true depth map, calculate the mean absolute error loss function based on the joint loss function and the training dataset, and calculate the total loss function based on the joint loss function and the mean absolute error loss function;

[0020] (8) Use the optimizer to optimize the deep alignment model on the training dataset, iterate several times, and use the deep alignment model to test on the test dataset until the difference between the output dataset of the deep alignment model and the 3D object detection label of the test dataset is less than the threshold. The deep alignment model training is complete.

[0021] Furthermore, the calculation steps for the joint loss function are as follows:

[0022] The predicted depth map for each image in the branch image is subtracted from the true depth map, and all elements are summed. Then, the sum is divided by the pixel value of the image to obtain the loss function for each image. The loss functions of each image are summed to obtain the joint loss function.

[0023] Furthermore, the calculation steps for the mean absolute error loss function are as follows:

[0024] The 3D object detection vector is subtracted from the 3D object detection labels in the training dataset. Then, the absolute value of each element is taken, and the average value of all elements is calculated to obtain the mean absolute error loss function.

[0025] A device for aligning 3D point clouds with dynamic depth maps of images, comprising:

[0026] The acquisition module is used to acquire point cloud data of the target from one viewpoint, images from multiple viewpoints, and the camera parameters corresponding to each image.

[0027] The computation module is used to input point cloud data, images, and camera parameters into the depth alignment model to obtain 3D object detection vectors;

[0028] The depth alignment model selects branch images using a gated network based on gated image features, obtains reference and source images based on the branch images, obtains a predicted depth map based on the reference and source images, obtains a true depth map based on the camera parameters and point cloud data corresponding to the reference image, and obtains a 3D target detection vector based on the image feature set of the gated image features, the true depth map, and the branch images.

[0029] Furthermore, the depth alignment model is characterized by being constructed through the following steps:

[0030] (1) Construct the format of the input dataset and the format of the output dataset;

[0031] (2) Construct a first convolutional layer for feature processing of images from multiple perspectives to obtain gated image features of images from multiple perspectives;

[0032] (3) Construct a gating network to convert the gating image features into probability vectors, and then find the images corresponding to the highest probability values ​​based on the probability vectors, i.e., branch images;

[0033] (4) Construct a second convolutional layer to obtain image feature sets for each image in the branch images, divide the branch images into reference images and source images, and obtain the predicted depth map through the stereo reconstruction network; obtain the real depth map based on the camera parameters corresponding to the reference image and the point cloud data;

[0034] (5) Tensor concatenation of the image feature sets of the branch images is performed to obtain multi-branch image features; tensor concatenation of the gated image features, the true depth map and the multi-branch image features is performed to obtain fused features; the fused features are input into the second convolutional layer to obtain output features; the output features are input into the global average pooling layer to obtain 3D object detection vectors;

[0035] (6) Input the training dataset and the test dataset into the depth alignment model;

[0036] (7) Calculate the joint loss function based on the predicted depth map and the true depth map, calculate the mean absolute error loss function based on the joint loss function and the training dataset, and calculate the total loss function based on the joint loss function and the mean absolute error loss function;

[0037] (8) Use the optimizer to optimize the deep alignment model on the training dataset, iterate several times, and use the deep alignment model to test on the test dataset until the difference between the output dataset of the deep alignment model and the 3D object detection label of the test dataset is less than the threshold. The deep alignment model training is complete.

[0038] An electronic device, comprising:

[0039] One or more processors;

[0040] Memory, used to store one or more programs;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0042] A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described herein.

[0043] The beneficial effects of this invention are as follows:

[0044] Leveraging the collaborative advantages of cross-view depth map estimation, depth map alignment is performed by sequentially selecting images from each viewpoint and point cloud data for depth map alignment, thereby enhancing the accuracy of depth map alignment estimation and resolving the projection distortion problem caused by sparse point clouds. Simultaneously, a dynamic gating network is constructed to determine which branches to select. The gating network can automatically decide to activate some branches from all image features based on the probability vector g; inactive branches do not participate in the computation, thus reducing computational load, improving operational efficiency, and alleviating the problem of excessive computational cost for multiple alignment branches corresponding to images from multiple viewpoints. Attached Figure Description

[0045] Figure 1 This is a flowchart of the method of the present invention;

[0046] Figure 2 This is a block diagram of the device of the present invention. Detailed Implementation

[0047] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0048] Leveraging the collaborative advantages of cross-view depth map estimation, depth map alignment is performed by sequentially selecting images from each viewpoint and point cloud data for depth map alignment, thereby enhancing the accuracy of depth map alignment estimation and resolving the projection distortion problem caused by sparse point clouds. Simultaneously, a dynamic gating network is constructed to determine which branches to select. This gating network can automatically decide to activate certain branches from all image features based on probability vectors, while inactive branches are not involved in the computation, thus reducing computational load, improving operational efficiency, and alleviating the problem of excessive computational cost for multiple alignment branches corresponding to images from multiple viewpoints.

[0049] The embodiments of the present invention will be further described below with reference to several examples.

[0050] Example 1

[0051] like Figure 1 A method for aligning 3D point clouds with dynamic depth maps of images includes the following steps:

[0052] S1. Collect point cloud data of the target from one viewpoint, images from multiple viewpoints, and camera parameters corresponding to each image.

[0053] Point cloud data is {P1, P2, ..., P} M}, where M is the total number of points in the point cloud data; images from multiple perspectives are {I1, I2, ..., I...} N}, where N is the total number of images, acquired by a camera from multiple perspectives. In this embodiment, the pixel count of a single image acquired by the camera is H×W; the camera parameters corresponding to each image are {S1, S2, ..., S}. N}={(K1,R1,T1)…,(K i ,R i ,T i ),…,(K N ,R N ,T N )}, where i is 1~N;

[0054] The point cloud data is generated by emitting laser pulses from a lidar onto the target surface. The coordinate system of the point cloud data is based on the scanning direction in front of the lidar as the X-axis, the left-hand direction perpendicular to the X-axis as the Y-axis, and the direction perpendicular to the X-axis and Y-axis as the Z-axis. The origin is the laser emission point.

[0055] The camera coordinate system has the optical center as the origin, the optical principal axis (shooting direction) as the Z-axis, the horizontal line of the imaging plane of the camera's image sensor as the X-axis, and the vertical line as the Y-axis.

[0056] Wherein, the intrinsic parameter matrix K i for:

[0057] ;

[0058] Where f i,x Let f be the horizontal pixel focal length of the i-th camera based on the camera coordinate system. i,y c is the vertical pixel focal length of the i-th camera based on the camera coordinate system; i,x Let c be the horizontal coordinate of the principal point of the i-th camera based on the camera coordinate system. i,y Let be the vertical coordinates of the principal point of the i-th camera based on the camera coordinate system;

[0059] Rotation matrix R i for:

[0060] ;

[0061] The elements represent the cosine of the angle between one axis of the camera coordinate system of the i-th camera and one axis of the point cloud data coordinate system. p1 is the x-axis of the camera coordinate system, p2 is the y-axis of the camera coordinate system, p3 is the z-axis of the camera coordinate system, q1 is the x-axis of the point cloud data coordinate system, q2 is the y-axis of the point cloud data coordinate system, and q3 is the z-axis of the point cloud data coordinate system, which are used for point cloud projection.

[0062] Translation vector T i for:

[0063] = ;

[0064] t i,x t i,y t i,z This represents the offset of the i-th camera along the x, y, and z axes of the camera coordinate system, used to describe the camera's pose with respect to the three-dimensional spatial coordinate system.

[0065] S2. Input point cloud data, images, and camera parameters into the depth alignment model to obtain a 3D target detection vector. The depth alignment model selects branch images using a gating network based on gating image features, obtains reference images and source images based on the branch images, obtains a predicted depth map based on the reference images and source images, obtains a true depth map based on the camera parameters corresponding to the reference images and point cloud data, and obtains a 3D target detection vector based on the image feature set of the gating image features, the true depth map, and the branch images.

[0066] The 3D target detection vector D = [x, y, z, w, h, d, θ], where x, y, z are the coordinates of the target's center point in the three-dimensional coordinate system, w, h, d are the width, height, and depth of the target, respectively; θ is the angle between the line connecting the target's center point and the origin of the three-dimensional coordinate system and the Z-axis of the three-dimensional coordinate system; the three-dimensional coordinate system has the city center where the target is located as the origin, the opposite direction of gravity as the Z-axis, the horizontal line as the X-axis, and the line perpendicular to the X-axis and Z-axis as the Y-axis.

[0067] The depth alignment model is constructed through the following steps:

[0068] (1) Construct the format of the input dataset and the format of the output dataset. The input dataset consists of point cloud data, images from multiple perspectives, and camera parameters corresponding to each image; the output dataset consists of 3D object detection vectors; thereby receiving the input data of the model and the output data of the model.

[0069] (2) Use the first convolutional layer of VGGnet to perform feature processing on images from multiple perspectives in the input dataset to obtain the gated image features (IF) of images from multiple perspectives;

[0070] (3) Construct a gating network, consisting of two layers:

[0071] 1) The first layer is the global average pooling layer of NINnet. This layer calculates the average value of all pixels in each feature map in the IF, turning each feature map into a numerical value, thereby converting the IF into a one-dimensional vector x.

[0072] 2) The second layer is the Softmax layer of NINnet, which transforms x into a probability vector g, as shown in the following formula:

[0073] g = Softmax(x);

[0074] 3) Traverse the probability vector g and find the graphs corresponding to the K highest probability values ​​in g, i.e., the branch graphs;

[0075] (4) Construct a single depth map alignment branch:

[0076] 1) For each image in the branch images, use the second convolutional layer of VGGnet for feature processing to obtain the image feature set B=[B1,B2,…,B…]. K From the branch images, select the j-th image as the reference image, where j is 1~K, and use the other images as source images; feed the image features corresponding to the reference image and source images into the stereo reconstruction network MVSnet, which sequentially predicts the corresponding predicted depth map of the reference image to obtain the predicted depth map M of all branch images. pred =[M 1pred ,…M j,pred …M K,pred ];

[0077] MVSnet predicts depth maps based on image features of a reference image, which provides information from multiple perspectives.

[0078] 2) Project the point cloud to obtain the true depth map:

[0079] The true depth map M is obtained based on the camera parameters corresponding to the reference image and the point cloud data. target =[M 1target ,…M j,target …M K,target ];

[0080] In this embodiment, a true depth map is obtained by projection using a pinhole camera model.

[0081] (5) Calculate the 3D target detection vector D:

[0082] The image feature set B of K branches is tensor concatenated to obtain the multi-branch image feature BA, which has the dimension of K×S×H×W.

[0083] Gated image features IF, M target And tensor concatenation of multi-branch image features BA to obtain fusion feature C, whose dimension is (S+K+K×S)×(H×W);

[0084] Input C into the second convolutional layer of VGGnet to obtain the output feature O, which has a dimension of 7×H×W;

[0085] Input O into the global average pooling layer of NINnet to obtain the 3D object detection vector D;

[0086] By using a learnable cross-viewpoint gating network, different viewpoint image combinations can be dynamically selected and aligned with the point cloud, thereby improving computational efficiency.

[0087] (6) Calculate the total loss function L of the depth alignment model:

[0088] M'(j)=M j,pred -M j,target ;

[0089] Add all elements of M'(j) to obtain M''(j);

[0090] L j =M''(j) / (H×W), where H is the pixel value in the height direction of the image and W is the pixel value in the width direction of the image;

[0091] The joint loss function is calculated as L all :

[0092] ;

[0093] Based on the 3D object detection vector D and the corresponding 3D object detection label D input during training. label =[x label ,y label ,z label ,w label ,h label ,d label ,θ label Calculate the mean absolute error loss function:

[0094] L MAE =1 / 7× (|x-x'|+|y-y'|+|z-z'|+|w-w'|+|h-h'|+|dd|'+|θ-θ'|);

[0095] According to L all and L MAE Calculate the total loss function L of the depth alignment model:

[0096] L=L MAE +L all ;

[0097] (7) Split the 72,000 data points into a training dataset and a test dataset in an 8:2 ratio. Input the training dataset and the test dataset into the depth alignment model. The training dataset and the test dataset are both composed of point cloud data, images from multiple perspectives, and camera parameters corresponding to each image, and are labeled with the corresponding 3D object detection label D. label =[x label ,y label ,z label ,wlabel ,h label ,d label ,θ label ];

[0098] The deep alignment model is optimized on the training dataset using the Adam optimizer for 100 iterations, and then tested on the test dataset using the same deep alignment model until the output dataset D of the deep alignment model matches the 3D object detection labels D of the test dataset. label If the difference is less than 5%, the deep alignment model training is complete.

[0099] Example 2

[0100] like Figure 2 A device for aligning 3D point clouds with dynamic depth maps of images, comprising:

[0101] The acquisition module is used to acquire point cloud data of the target from one viewpoint, images from multiple viewpoints, and the camera parameters corresponding to each image.

[0102] The calculation module is used to input point cloud data, images, and camera parameters into the depth alignment model to obtain a 3D target detection vector D=[x,y,z,w,h,d,θ], where x,y,z are the coordinates of the target's center point in the three-dimensional coordinate system, w,h,d are the width, height, and depth of the target, respectively, and θ is the angle between the line connecting the target's center point and the origin of the three-dimensional coordinate system and the Z-axis of the three-dimensional coordinate system.

[0103] The method for constructing the depth alignment model is the same as that in Example 1.

[0104] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0105] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0106] Accordingly, this application also provides an electronic device, including:

[0107] An electronic device, comprising:

[0108] One or more processors;

[0109] Memory, used to store one or more programs;

[0110] When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0111] Accordingly, this application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the above methods.

[0112] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can also be implemented in other ways. The method and system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0113] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0114] On the other hand, a computer-readable storage medium stores computer instructions thereon, which, when executed by a processor, implement the steps of the above-described method. When the computer program is executed by the processor, it implements the method as described in any of the first aspects above. If the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0115] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for aligning 3D point clouds with dynamic depth maps of images, characterized in that, Includes the following steps: S1. Collect point cloud data of the target from one viewpoint, images from multiple viewpoints, and camera parameters corresponding to each image. S2. Input the point cloud data, image, and camera parameters into the depth alignment model to obtain the 3D object detection vector; The depth alignment model selects branch images using a gated network based on gated image features, obtains reference and source images based on the branch images, obtains a predicted depth map based on the reference and source images, obtains a true depth map based on the camera parameters and point cloud data corresponding to the reference image, and obtains a 3D target detection vector based on the image feature set of the gated image features, the true depth map, and the branch image. The depth alignment model is constructed through the following steps: (1) Construct a first convolutional layer for feature processing of images from multiple perspectives to obtain gated image features of images from multiple perspectives; (2) Construct a gating network to convert the gating image features into probability vectors, and then find the images corresponding to the highest probability values ​​based on the probability vectors, i.e., branch images; (3) Construct a second convolutional layer to obtain image feature sets based on each image in the branch images, divide the branch images into reference images and source images, and obtain the predicted depth map through a stereo reconstruction network; The true depth map is obtained based on the camera parameters and point cloud data corresponding to the reference image; (4) Tensor concatenation of the image feature sets of the branch images is performed to obtain multi-branch image features; tensor concatenation of the gated image features, the true depth map and the multi-branch image features is performed to obtain fused features; the fused features are input into the second convolutional layer to obtain output features; the input is input into the global average pooling layer to obtain 3D object detection vector.

2. The method according to claim 1, characterized in that, The depth alignment model is trained through the following steps: (1) Construct the format of the input dataset and the format of the output dataset; (2) Input the training dataset and the test dataset into the depth alignment model; (3) Calculate the joint loss function based on the predicted depth map and the true depth map, calculate the mean absolute error loss function based on the joint loss function and the training dataset, and calculate the total loss function based on the joint loss function and the mean absolute error loss function; (4) Use the optimizer to optimize the deep alignment model on the training dataset, iterate several times, and use the deep alignment model to test on the test dataset until the difference between the output dataset of the deep alignment model and the 3D object detection label of the test dataset is less than the threshold. The deep alignment model training is complete.

3. The method according to claim 2, characterized in that, The calculation steps for the joint loss function are as follows: The predicted depth map for each image in the branch image is subtracted from the true depth map, and all elements are summed. Then, the sum is divided by the pixel value of the image to obtain the loss function for each image. The loss functions of each image are summed to obtain the joint loss function.

4. The method according to claim 2, characterized in that, The calculation steps for the mean absolute error loss function are as follows: The 3D object detection vector is subtracted from the 3D object detection labels in the training dataset. Then, the absolute value of each element is taken, and the average value of all elements is calculated to obtain the mean absolute error loss function.

5. A device for aligning 3D point clouds with dynamic depth maps of images, characterized in that, include: The acquisition module is used to acquire point cloud data of the target from one viewpoint, images from multiple viewpoints, and the camera parameters corresponding to each image. The computation module is used to input point cloud data, images, and camera parameters into the depth alignment model to obtain 3D object detection vectors; The depth alignment model selects branch images using a gated network based on gated image features, obtains reference and source images based on the branch images, obtains a predicted depth map based on the reference and source images, obtains a true depth map based on the camera parameters and point cloud data corresponding to the reference image, and obtains a 3D target detection vector based on the image feature set of the gated image features, the true depth map, and the branch image. The depth alignment model is constructed through the following steps: (1) Construct a first convolutional layer for feature processing of images from multiple perspectives to obtain gated image features of images from multiple perspectives; (2) Construct a gating network to convert the gating image features into probability vectors, and then find the images corresponding to the highest probability values ​​based on the probability vectors, i.e., branch images; (3) Construct a second convolutional layer to obtain image feature sets based on each image in the branch images, divide the branch images into reference images and source images, and obtain the predicted depth map through a stereo reconstruction network; The true depth map is obtained based on the camera parameters and point cloud data corresponding to the reference image; (4) Tensor concatenation of the image feature sets of the branch images is performed to obtain multi-branch image features; tensor concatenation of the gated image features, the true depth map and the multi-branch image features is performed to obtain fused features; the fused features are input into the second convolutional layer to obtain output features; the input is input into the global average pooling layer to obtain 3D object detection vector.

6. The apparatus according to claim 5, characterized in that, The depth alignment model is constructed through the following steps: (1) Construct the format of the input dataset and the format of the output dataset; (2) Input the training dataset and the test dataset into the depth alignment model; (3) Calculate the joint loss function based on the predicted depth map and the true depth map, calculate the mean absolute error loss function based on the joint loss function and the training dataset, and calculate the total loss function based on the joint loss function and the mean absolute error loss function; (4) Use the optimizer to optimize the deep alignment model on the training dataset, iterate several times, and use the deep alignment model to test on the test dataset until the difference between the output dataset of the deep alignment model and the 3D object detection label of the test dataset is less than the threshold. The deep alignment model training is complete.

7. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-4.

Images