Multi-sensor fusion three-dimensional reconstruction method, device and equipment and storage medium

Abstract

The invention discloses a multi-sensor fusion three-dimensional reconstruction method, device and equipment and a storage medium. The method comprises the following steps: acquiring image data and point cloud data through sensors; and inputting the obtained image data and point cloud data into the trained three-dimensional reconstruction model to generate a point-by-point RGB prediction result so as to realize reconstruction of a three-dimensional scene. According to the method, image features are input into a cross attention module to serve as keys and values, point cloud features are used as query deaggregated image features, and the point cloud features are updated; the point cloud features are processed through a reconstruction prediction head, a point-by-point RGB prediction result is generated, and reconstruction of a three-dimensional scene is achieved; calibration information and model input are decoupled, implicit space alignment of multi-sensor information is achieved, and robustness of calibration noise is improved. The method can be widely applied to the technical field of intelligent identification.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition technology, and in particular to a multi-sensor fusion three-dimensional reconstruction method, device, equipment and storage medium. Background Art

[0002] Three-dimensional reconstruction plays an important role in the field of autonomous driving. It provides vehicles with geometric and appearance information of the surrounding environment. In order to achieve comprehensive and accurate environmental perception, autonomous vehicles are usually equipped with multiple sensors, such as cameras and lidar. The RGB images output by the camera have rich texture and color information, but lack geometric perception and are easily disturbed by lighting conditions. The three-dimensional point cloud output by the lidar has accurate spatial information and strong anti-interference ability, but lacks appearance information. Therefore, using the complementary information of cameras and lidars to achieve multi-sensor fusion three-dimensional reconstruction has gradually become a research hotspot.

[0003] Existing multi-sensor fusion methods explicitly rely on the calibration information of sensors. They use the calibration information to construct the mapping relationship between image and point cloud data, ensure the consistency between different sensor data, and further fuse the aligned multi-sensor data. However, when the different sensor data lack good alignment, these methods will suffer from serious performance degradation, that is, they are easily disturbed by calibration noise. However, whether in the joint calibration process of multiple sensors or in the actual application process of multiple sensors, calibration noise is difficult to completely eliminate, which greatly limits the application of existing methods in real scenarios. Therefore, in practical applications, it is crucial for multi-sensor fusion methods to be robust to calibration noise. However, how to achieve implicit spatial alignment of multi-sensor information without explicitly relying on calibration information remains to be solved. Summary of the invention

[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the object of the present invention is to provide a multi-sensor fusion 3D reconstruction method, device, equipment and storage medium with robust calibration information.

[0005] The first technical solution adopted by the present invention is:

[0006] A multi-sensor fusion 3D reconstruction method comprises the following steps:

[0007] Acquire image data and point cloud data through sensors;

[0008] The obtained image data and point cloud data are input into the trained 3D reconstruction model to generate point-by-point RGB prediction results to achieve reconstruction of the 3D scene;

[0009] The working method of the three-dimensional reconstruction model is as follows:

[0010] Perform feature extraction on the input image data to obtain image features;

[0011] Perform feature extraction on the input point cloud data to obtain point-by-point features;

[0012] Use the point-by-point features as queries for the cross-attention module, perform feature aggregation on the image features, and update the point-by-point features;

[0013] The updated point-by-point features are input into the reconstruction prediction head to generate point-by-point RGB prediction results.

[0014] Furthermore, the 3D reconstruction model is a 3D reconstruction model based on Transformer, and the 3D reconstruction model is trained in the following manner:

[0015] Build a standard multi-sensor truth system and set it on the vehicle to provide aligned RGB-XYZ truth information;

[0016] Acquiring image data through a first sensor, and acquiring point cloud data through a second sensor;

[0017] Build a 3D reconstruction model, input the image data into the 3D reconstruction model as the key and value of the Transformer, use the point cloud data to query, predict the RGB information of the corresponding XYZ coordinates, and obtain the point-by-point RGB prediction results;

[0018] The reconstruction loss is calculated based on the aligned RGB-XYZ true value information and the point-by-point RGB prediction results, and the 3D reconstruction model is trained based on the loss.

[0019] Furthermore, the construction of a standard multi-sensor truth value system is arranged on the vehicle to provide aligned RGB-XYZ truth value information, including:

[0020] According to the projection matrix of the camera and the laser radar in the truth system, the point-by-point three-dimensional coordinates (x, y, z) of the point cloud data are mapped to obtain the pixel coordinates (u, v) on the image data;

[0021] The corresponding RGB information of the image data is obtained through the pixel coordinate index, and the RGB information of the image is mapped to the point cloud space to obtain paired RGB-XYZ true value information for supervised training of the 3D reconstruction model.

[0022] Furthermore, the three-dimensional reconstruction model includes an image encoder, a point cloud encoder, a cross attention module and a reconstruction prediction head;

[0023] The image encoder is used to extract features from input image data to obtain image features;

[0024] The point cloud encoder is used to extract features from the input point cloud data to obtain point-by-point features;

[0025] The cross attention module is used to perform feature aggregation on image features and update point-by-point features;

[0026] The reconstruction prediction head is used to generate point-by-point RGB prediction results according to the updated point-by-point features to achieve reconstruction of the three-dimensional scene.

[0027] Furthermore, the image data is input into the 3D reconstruction model as the key and value of the Transformer, the point cloud data is used for query, the RGB information of the corresponding XYZ coordinates is predicted, and the point-by-point RGB prediction results are obtained, including:

[0028] The image features are input into the cross-attention module as keys and values, and the image features are dynamically queried using the point cloud features; wherein the updated point cloud features contain geometric information and texture information;

[0029] Input the updated point cloud features into the reconstruction prediction head to generate the point cloud RGB prediction results

[0030] Furthermore, the calculation formula of the reconstruction loss is as follows:

[0031]

[0032] Where SmoothL1 represents the reconstruction loss function, and rgb Represent the point-by-point RGB prediction results and point-by-point RGB labels respectively.

[0033] Furthermore, the multi-sensor fusion 3D reconstruction method further includes the following steps:

[0034] After the three-dimensional reconstruction model training is completed, the true value system is deleted.

[0035] Furthermore, the first sensor is a vehicle-mounted camera, and the second sensor is a laser radar.

[0036] The second technical solution adopted by the present invention is:

[0037] A multi-sensor fusion three-dimensional reconstruction device, comprising:

[0038] A data acquisition module, used to acquire image data and point cloud data through sensors;

[0039] The data processing module is used to input the acquired image data and point cloud data into the trained 3D reconstruction model to generate point-by-point RGB prediction results to achieve reconstruction of the 3D scene;

[0040] The working method of the three-dimensional reconstruction model is as follows:

[0041] Perform feature extraction on the input image data to obtain image features;

[0042] Perform feature extraction on the input point cloud data to obtain point-by-point features;

[0043] Use the point-by-point features as queries for the cross-attention module, perform feature aggregation on the image features, and update the point-by-point features;

[0044] The updated point-by-point features are input into the reconstruction prediction head to generate point-by-point RGB prediction results.

[0045] The third technical solution adopted by the present invention is:

[0046] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a multi-sensor fusion three-dimensional reconstruction method as described above.

[0047] The fourth technical solution adopted by the present invention is:

[0048] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a multi-sensor fusion three-dimensional reconstruction method as described above.

[0049] The fifth technical solution adopted by the present invention is:

[0050] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned multi-sensor fusion three-dimensional reconstruction method.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention inputs image features into the cross-attention module as keys and values, uses point cloud features as queries to de-aggregate image features and update point cloud features; processes point cloud features through a reconstruction prediction head to generate point-by-point RGB prediction results to achieve reconstruction of three-dimensional scenes. Unlike existing methods that explicitly rely on calibration information, the present invention decouples calibration information from model input, achieves implicit spatial alignment of multi-sensor information, and improves robustness to calibration noise. In actual applications of autonomous driving scenarios, autonomous driving vehicles build a standard truth system, provide calibrated RGB-XYZ information, and use sensor information installed on the vehicle as input. Based on the method, the model can directly reconstruct spatial information consistent with the truth system from the sensor information on the vehicle, thereby avoiding calibration of each vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 is a flowchart of the steps of a multi-sensor fusion 3D reconstruction method in an embodiment of the present invention;

[0054] Figure 2 is a workflow diagram of a three-dimensional reconstruction model in an embodiment of the present invention;

[0055] Figure 3 4 is a training flow chart of a three-dimensional reconstruction model of a three-dimensional reconstruction model in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0057] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0058] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0059] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention in combination with the specific content of the technical solution.

[0060] Example 1

[0061] like Figure 1 As shown, this embodiment provides a multi-sensor fusion 3D reconstruction method with robust calibration information, comprising the following steps:

[0062] S1. Obtain image data and point cloud data through sensors.

[0063] Specifically, the image data is obtained through the vehicle-mounted camera, and the point cloud data is obtained through the laser radar. The image data output by the camera H and W represent the height and width respectively, and each pixel contains three color channels, RGB. Point cloud data output by the LiDAR Where N represents the number of point clouds, and each point p∈P contains three-dimensional coordinates (x, y, z) and reflection intensity i.

[0064] S2. Input the obtained image data and point cloud data into the trained 3D reconstruction model to generate point-by-point RGB prediction results to achieve reconstruction of the 3D scene.

[0065] As an optional implementation, the 3D reconstruction model is a 3D reconstruction model based on Transformer, and the 3D reconstruction model is constructed by:

[0066] Using convolutional layers, batch normalization layers, linear rectification layers, pooling layers, and linear layers, a Transformer-based 3D reconstruction model is constructed, including an image encoder, a point cloud encoder, a cross-attention module, and a reconstruction prediction head.

[0067] join Figure 2 , the 3D reconstruction model works as follows:

[0068] A1. Generate image features and point cloud features through image encoder and point cloud encoder respectively.

[0069] The image data output by the camera is input into the image encoder to obtain image features with rich texture information. To promote the learning of implicit spatial alignment of multi-sensor data, the point cloud data output by the lidar is first subjected to appropriate pose transformation, and then input into the point cloud encoder to obtain point cloud features with rich geometric information. If the resolution of the input image is too high, the resolution can be adjusted appropriately; if the number of input point clouds is too large, it can be appropriately downsampled.

[0070] A2. Learning implicit spatial alignment between image and point cloud data via a criss-cross attention module.

[0071] The image features are input into the cross-attention module as keys and values, and the point cloud features are used to dynamically query the image features. The updated point cloud features contain geometric and texture information, which can well represent the 3D scene.

[0072] A3. Generate point cloud RGB prediction results by reconstructing the prediction head.

[0073] As an optional implementation, see Figure 3 , the 3D reconstruction model is trained in the following way:

[0074] B1. Build a standard multi-sensor truth system and set it on the vehicle to provide aligned RGB-XYZ truth information.

[0075] Jointly calibrate multiple sensors and build a standard multi-sensor truth system. Specifically, according to the projection matrix of the camera and lidar in the truth system, the point-by-point 3D coordinates (x, y, z) of the point cloud data are mapped to the pixel coordinates (u, v) on the image data, and the corresponding RGB information on the image data is obtained through the pixel coordinate index, thereby mapping the image RGB information to the point cloud space to obtain paired RGB-XYZ information, which is used as a supervisory signal for the training of the 3D reconstruction model.

[0076] B2. Obtain image data through the vehicle-mounted camera and obtain point cloud data through the lidar.

[0077] Specifically, the image data output by the camera H and W represent the height and width respectively, and each pixel contains three color channels, RGB. Point cloud data output by the LiDAR Where N represents the number of point clouds, and each point p∈P contains three-dimensional coordinates (x, y, z) and reflection intensity i.

[0078] B3. Input the image data into the 3D reconstruction model as the key and value of the Transformer, use the point cloud data for query, predict the RGB information of the corresponding XYZ coordinates, and obtain the point-by-point RGB prediction results.

[0079] In some embodiments, step B3 specifically includes:

[0080] B31, using an image encoder to extract features from the input image data to obtain image features;

[0081] B32. Use the point cloud encoder to extract features from the input point cloud data to obtain point-by-point features;

[0082] B33, using the point-by-point features as queries for the cross-attention module, performing feature aggregation on the image features, and updating the point-by-point features;

[0083] B34. Input the updated point-by-point features into the reconstruction prediction head to generate point-by-point RGB prediction results to achieve reconstruction of the three-dimensional scene.

[0084] B4. Calculate the reconstruction loss based on the aligned RGB-XYZ true value information and the point-by-point RGB prediction results, and train the 3D reconstruction model based on the loss.

[0085] Input the updated point cloud features into the reconstruction prediction head to generate the point cloud RGB prediction results Realize the reconstruction of the three-dimensional scene. According to the aligned RGB-XYZ supervision signal O rgb And the reconstruction loss function SmoothL1, calculate the reconstruction loss Used for model training.

[0086] The trained 3D reconstruction model uses the uncalibrated vehicle sensor information as input to reconstruct the 3D scene in the true value system coordinate system. The trained 3D reconstruction model has the ability to implicitly align multi-sensor information in space, does not rely on the calibration information of multiple sensors, decouples the calibration information from the model input, and uses the uncalibrated vehicle sensor information as input to reconstruct the 3D scene in the true value system coordinate system, thereby avoiding the need to calibrate each vehicle.

[0087] In summary, the method of this embodiment is different from the existing method that explicitly relies on calibration information. It decouples the calibration information from the model input, realizes implicit spatial alignment of multi-sensor information, and improves the robustness to calibration noise. In the actual application of autonomous driving scenarios, the autonomous driving vehicle builds a standard truth system, provides calibrated RGB-XYZ information, and uses the sensor information installed on the vehicle as input. Based on the method, the model can directly reconstruct spatial information consistent with the truth system from the sensor information on the vehicle, thereby avoiding calibration of each vehicle.

[0088] Example 2

[0089] This embodiment provides a multi-sensor fusion 3D reconstruction device, including:

[0090] A data acquisition module, used to acquire image data and point cloud data through sensors;

[0091] The data processing module is used to input the acquired image data and point cloud data into the trained 3D reconstruction model to generate point-by-point RGB prediction results to achieve reconstruction of the 3D scene;

[0092] The working method of the three-dimensional reconstruction model is as follows:

[0093] Perform feature extraction on the input image data to obtain image features;

[0094] Perform feature extraction on the input point cloud data to obtain point-by-point features;

[0095] Use the point-by-point features as queries for the cross-attention module, perform feature aggregation on the image features, and update the point-by-point features;

[0096] The updated point-by-point features are input into the reconstruction prediction head to generate point-by-point RGB prediction results.

[0097] Since the device is a multi-sensor fusion three-dimensional reconstruction device of an embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0098] Example 3

[0099] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 1 A multi-sensor fusion 3D reconstruction method is shown.

[0100] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0101] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0102] Since the electronic device is an electronic device corresponding to a multi-sensor fusion three-dimensional reconstruction method of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0103] Example 4

[0104] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 A multi-sensor fusion 3D reconstruction method is shown.

[0105] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0106] Since the storage medium is a storage medium corresponding to a multi-sensor fusion three-dimensional reconstruction method of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0107] Example 5

[0108] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a multi-sensor fusion three-dimensional reconstruction method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0109] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0110] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0111] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-sensor fusion 3D reconstruction method, characterized in that: The following steps are involved: Acquire image data and point cloud data through sensors; The obtained image data and point cloud data are input into the trained 3D reconstruction model to generate point-by-point RGB prediction results to achieve reconstruction of the 3D scene; The working method of the three-dimensional reconstruction model is as follows: Perform feature extraction on the input image data to obtain image features; Perform feature extraction on the input point cloud data to obtain point-by-point features; Use the point-by-point features as queries for the cross-attention module, perform feature aggregation on the image features, and update the point-by-point features; The updated point-by-point features are input into the reconstruction prediction head to generate point-by-point RGB prediction results.

2. The multi-sensor fusion 3D reconstruction method according to claim 1, characterized in that: The 3D reconstruction model is a Transformer-based 3D reconstruction model, and the 3D reconstruction model is trained in the following manner: Build a standard multi-sensor truth value system and set it on the vehicle to provide RGB-XYZ truth value information; Acquiring image data through a first sensor, and acquiring point cloud data through a second sensor; Build a 3D reconstruction model, input the image data into the 3D reconstruction model as the key and value of the Transformer, use the point cloud data to query, predict the RGB information of the corresponding XYZ coordinates, and obtain the point-by-point RGB prediction results; The reconstruction loss is calculated based on the RGB-XYZ true value information and the point-by-point RGB prediction results, and the 3D reconstruction model is trained based on the loss.

3. The multi-sensor fusion 3D reconstruction method according to claim 2, characterized in that: The construction of a standard multi-sensor truth value system is set on the vehicle to provide aligned RGB-XYZ truth value information, including: According to the projection matrix of the camera and the laser radar in the truth system, the point-by-point three-dimensional coordinates (x, y, z) of the point cloud data are mapped to obtain the pixel coordinates (u, v) on the image data; The corresponding RGB information on the image data is obtained through the pixel coordinate index, and the RGB information of the image is mapped to the point cloud space to obtain the RGB-XYZ true value information for supervised training of the 3D reconstruction model.

4. A multi-sensor fusion 3D reconstruction method according to claim 1 or 2, characterized in that: The three-dimensional reconstruction model includes an image encoder, a point cloud encoder, a cross attention module and a reconstruction prediction head; The image encoder is used to extract features from input image data to obtain image features; The point cloud encoder is used to extract features from the input point cloud data to obtain point-by-point features; The cross attention module is used to perform feature aggregation on image features and update point-by-point features; The reconstruction prediction head is used to generate point-by-point RGB prediction results according to the updated point-by-point features to achieve reconstruction of the three-dimensional scene.

5. The multi-sensor fusion 3D reconstruction method according to claim 4, characterized in that: The image data is input into the 3D reconstruction model as the key and value of the Transformer, and the point cloud data is used for querying to predict the RGB information of the corresponding XYZ coordinates to obtain the point-by-point RGB prediction results, including: The image features are input into the cross-attention module as keys and values, and the image features are dynamically queried using the point cloud features; wherein the updated point cloud features contain geometric information and texture information; Input the updated point cloud features into the reconstruction prediction head to generate the point cloud RGB prediction results 6. The multi-sensor fusion 3D reconstruction method according to claim 2, characterized in that: The calculation formula of the reconstruction loss is as follows: Where SmoothL1 represents the reconstruction loss function, and rgb Represent the point-by-point RGB prediction results and point-by-point RGB labels respectively.

7. The multi-sensor fusion 3D reconstruction method according to claim 2, characterized in that: The multi-sensor fusion 3D reconstruction method also includes the following steps: After the three-dimensional reconstruction model training is completed, the true value system is deleted.

8. A multi-sensor fusion 3D reconstruction device, characterized in that: include: A data acquisition module, used to acquire image data and point cloud data through sensors; The data processing module is used to input the acquired image data and point cloud data into the trained 3D reconstruction model to generate point-by-point RGB prediction results to achieve reconstruction of the 3D scene; The working method of the three-dimensional reconstruction model is as follows: Perform feature extraction on the input image data to obtain image features; Perform feature extraction on the input point cloud data to obtain point-by-point features; Use the point-by-point features as queries for the cross-attention module, perform feature aggregation on the image features, and update the point-by-point features; The updated point-by-point features are input into the reconstruction prediction head to generate point-by-point RGB prediction results.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a multi-sensor fusion three-dimensional reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a multi-sensor fusion three-dimensional reconstruction method as described in any one of claims 1 to 7.

Images