A multi-sensor fusion three-dimensional reconstruction method, device, equipment and storage medium

Abstract

The application discloses a kind of multi-sensor fusion three-dimensional reconstruction method, device, equipment and storage medium, wherein method includes the following steps: image data and point cloud data are obtained by sensor;The image data and point cloud data obtained are input into the three-dimensional reconstruction model trained, generate point-by-point RGB prediction result, to realize the reconstruction of three-dimensional scene.The application inputs image features into cross attention module as key and value, uses point cloud features as query to aggregate image features and update point cloud features;Point cloud features are processed by reconstruction prediction head, generate point-by-point RGB prediction result, realize the reconstruction of three-dimensional scene;Calibration information is decoupled with model input, realizes the implicit spatial alignment of multi-sensor information, improves the robustness to calibration noise.The application can be widely applied in intelligent identification technical field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent identification, and in particular to a multi-sensor fusion three-dimensional reconstruction method, device, equipment and storage medium. BACKGROUND

[0002] Three-dimensional reconstruction plays an important role in the field of autonomous driving, which provides vehicles with geometric and appearance information of the surrounding environment. In order to achieve comprehensive and accurate environmental perception, autonomous vehicles are usually equipped with multiple sensors such as cameras and lidars. The RGB image output by the camera has rich texture and color information, but lacks geometric perception and is easily disturbed by lighting conditions. The three-dimensional point cloud output by the lidar has accurate spatial information and strong anti-interference ability, but lacks appearance information. Therefore, using the complementary information of cameras and lidars to realize multi-sensor fusion three-dimensional reconstruction has gradually become a research hotspot.

[0003] Existing multi-sensor fusion methods explicitly rely on the calibration information of the sensors. They construct the mapping relationship between the image and the point cloud data through the calibration information to ensure the consistency between different sensor data, and further fuse the aligned multi-sensor data. However, when different sensor data lack good alignment, these methods will have serious performance degradation, that is, they are easily disturbed by calibration noise. However, calibration noise is difficult to completely eliminate in the joint calibration process of multi-sensor or in the actual application process of multi-sensor, which will greatly limit the application of existing methods in real scenes. Therefore, in actual application, it is crucial for multi-sensor fusion methods to be robust to calibration noise, but how to realize implicit spatial alignment of multi-sensor information without explicitly relying on calibration information still needs to be solved. SUMMARY

[0004] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a multi-sensor fusion three-dimensional reconstruction method, device, equipment and storage medium which is robust to calibration information.

[0005] The first technical solution adopted by the present application is:

[0006] A multi-sensor fusion three-dimensional reconstruction method, comprising the following steps:

[0007] obtaining image data and point cloud data through sensors;

[0008] inputting the obtained image data and point cloud data into a trained three-dimensional reconstruction model to generate a point-by-point RGB prediction result, so as to realize reconstruction of a three-dimensional scene;

[0009] wherein the working mode of the three-dimensional reconstruction model is as follows:

[0010] perform feature extraction on the input image data to obtain image features;

[0011] perform feature extraction on the input point cloud data to obtain point-wise features;

[0012] use the point-wise features as queries of a cross-attention module, perform feature aggregation on the image features, and update the point-wise features;

[0013] input the updated point-wise features into a reconstruction prediction head to generate point-wise RGB prediction results.

[0014] Further, the three-dimensional reconstruction model is a three-dimensional reconstruction model based on a Transformer, and the three-dimensional reconstruction model is trained in the following manner:

[0015] a set of standard multi-sensor ground truth systems is constructed and arranged on a vehicle to provide aligned RGB-XYZ ground truth information;

[0016] acquire image data through a first sensor and acquire point cloud data through a second sensor;

[0017] construct a three-dimensional reconstruction model, input the image data into the three-dimensional reconstruction model as keys and values of a Transformer, use the point cloud data for querying, predict RGB information of corresponding XYZ coordinates, and obtain point-wise RGB prediction results;

[0018] calculate a reconstruction loss according to the aligned RGB-XYZ ground truth information and the point-wise RGB prediction results, and train the three-dimensional reconstruction model according to the loss.

[0019] Further, the set of standard multi-sensor ground truth systems arranged on the vehicle to provide aligned RGB-XYZ ground truth information comprises:

[0020] According to the projection matrix of the camera and the lidar in the ground truth system, the point-wise three-dimensional coordinates (x, y, z) of the point cloud data are mapped to obtain pixel coordinates (u, v) on the image data;

[0021] obtain corresponding RGB information on the image data through the pixel coordinate index, map the RGB information of the image to the point cloud space, and obtain paired RGB-XYZ ground truth information for supervised training of the three-dimensional reconstruction model.

[0022] Further, the three-dimensional reconstruction model comprises an image encoder, a point cloud encoder, a cross-attention module, and a reconstruction prediction head;

[0023] The image encoder is configured to perform feature extraction on the input image data to obtain image features;

[0024] The point cloud encoder is used for feature extraction on input point cloud data to obtain point-by-point features;

[0025] The cross attention module is used for feature aggregation on image features and updating point-by-point features;

[0026] The reconstruction prediction head is used for generating point-by-point RGB prediction results according to the updated point-by-point features, so as to realize reconstruction of a three-dimensional scene.

[0027] Further, the image data is input into the three-dimensional reconstruction model as keys and values of the Transformer, the point cloud data is used for query, RGB information of corresponding XYZ coordinates is predicted, and point-by-point RGB prediction results are obtained, including:

[0028] The image features are input into the cross attention module as keys and values, and the point cloud features are used for dynamic query of the image features; wherein the updated point cloud features contain geometric information and texture information;

[0029] The updated point cloud features are input into the reconstruction prediction head to generate point cloud RGB prediction results

[0030] Further, the calculation formula of the reconstruction loss is as follows:

[0031]

[0032] In the formula, SmoothL1 represents a reconstruction loss function, and o rgb respectively represent point-by-point RGB prediction results and point-by-point RGB labels.

[0033] Further, the multi-sensor fusion three-dimensional reconstruction method further includes the following steps:

[0034] After the three-dimensional reconstruction model is trained, the true value system is deleted.

[0035] Further, the first sensor is a vehicle-mounted camera, and the second sensor is a laser radar.

[0036] The second technical solution adopted by the present application is:

[0037] A multi-sensor fusion three-dimensional reconstruction device comprises:

[0038] A data acquisition module is used for acquiring image data and point cloud data through a sensor;

[0039] A data processing module is used for inputting the obtained image data and point cloud data into a trained three-dimensional reconstruction model to generate point-by-point RGB prediction results, so as to realize reconstruction of a three-dimensional scene.

[0040] The working mode of the three-dimensional reconstruction model is as follows:

[0041] Feature extraction is performed on the input image data to obtain image features;

[0042] Feature extraction is performed on the input point cloud data to obtain point-by-point features;

[0043] The point-by-point features are used as queries for a cross-attention module, the image features are aggregated, and the point-by-point features are updated;

[0044] The updated point-by-point features are input into a reconstruction prediction head to generate point-by-point RGB prediction results.

[0045] The third technical solution adopted by the present application is:

[0046] An electronic device, comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement a multi-sensor fusion three-dimensional reconstruction method as described above.

[0047] The fourth technical solution adopted by the present application is:

[0048] A computer-readable storage medium, the storage medium storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement a multi-sensor fusion three-dimensional reconstruction method as described above.

[0049] The fifth technical solution adopted by the present application is:

[0050] A computer program product or computer program, the computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to make the computer device execute the above-mentioned multi-sensor fusion three-dimensional reconstruction method.

[0051] Compared with the prior art, the present application has the beneficial effect that the present application inputs image features into a cross-attention module as keys and values, uses point cloud features as queries to aggregate image features and update point cloud features, processes point cloud features through a reconstruction prediction head to generate point-by-point RGB prediction results, and realizes reconstruction of a three-dimensional scene. The present application differs from existing methods in that it explicitly relies on calibration information, decouples calibration information from model input, realizes implicit spatial alignment of multi-sensor information, and improves robustness to calibration noise. In practical applications in the automatic driving scene, an automatic driving vehicle constructs a standard ground truth system, provides calibrated RGB-XYZ information, uses sensor information equipped on the vehicle as input, and enables the model to reconstruct spatial information consistent with the ground truth system based on the method, thereby avoiding calibration for each vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments in the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0053] Figure 1 is a step flow chart of a multi-sensor fusion three-dimensional reconstruction method in an embodiment of the present application;

[0054] Figure 2 is a work flow chart of a three-dimensional reconstruction model in an embodiment of the present application;

[0055] Figure 3 is a training flow chart of a three-dimensional reconstruction model of a three-dimensional reconstruction model in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0057] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by the upper, lower, front, rear, left, right and the like, is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0058] In the description of the present application, the meaning of one or more is one or more, the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, etc. are understood as including the number. If the first, second is described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of technical features indicated.

[0059] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0060] Embodiment 1

[0061] As shown in the following figure, the present embodiment provides a multi-sensor fusion three-dimensional reconstruction method robust to calibration information, including the following steps: Figure 1

[0062] S1, obtaining image data and point cloud data through a sensor.

[0063] Specifically, the image data is obtained through a vehicle-mounted camera, and the point cloud data is obtained through a laser radar. The image data output by the camera contains three color channels of RGB, and H and W represent the height and width respectively. The point cloud data output by the laser radar contains three-dimensional coordinates (x, y, z) and reflection intensity i for each point p∈P. The point cloud data output by the laser radar contains three-dimensional coordinates (x, y, z) and reflection intensity i for each point p∈P.

[0064] S2, inputting the obtained image data and point cloud data into a trained three-dimensional reconstruction model to generate a point-by-point RGB prediction result, so as to realize the reconstruction of a three-dimensional scene.

[0065] As an optional implementation, the three-dimensional reconstruction model is a three-dimensional reconstruction model based on Transformer, which is obtained by the following way:

[0066] ​A three-dimensional reconstruction model based on the Transformer is constructed by using convolutional layers, batch normalization layers, linear rectifier layers, pooling layers, and linear layers, including an image encoder, a point cloud encoder, a cross-attention module, and a reconstruction prediction head.

[0067] participate Figure 2 The three-dimensional reconstruction model works as follows:

[0068] A1. Image features and point cloud features are respectively generated by the image encoder and the point cloud encoder.

[0069] Image data output by the camera is input into the image encoder to obtain image features with rich texture information. To facilitate the learning of implicit spatial alignment of multi-sensor data, appropriate pose transformation is first applied to the point cloud data output by the lidar, and then the point cloud data is input into the point cloud encoder to obtain point cloud features with rich geometric information. If the resolution of the input image is too high, the resolution can be adjusted appropriately; if the number of input point clouds is too large, the point clouds can be down-sampled appropriately.

[0070] A2. The cross-attention module is used to learn the implicit spatial alignment between the image and the point cloud data.

[0071] The image features are input into the cross-attention module as keys and values, and the point cloud features are used to dynamically query the image features. The updated point cloud features contain both geometric information and texture information, and can well represent a three-dimensional scene.

[0072] A3. The point cloud RGB prediction result is generated by the reconstruction prediction head.

[0073] As an optional implementation, see Figure 3 The three-dimensional reconstruction model is trained in the following manner:

[0074] B1. A standard multi-sensor ground truth system is constructed on a vehicle to provide aligned RGB-XYZ ground truth information.

[0075] The multi-sensor is jointly calibrated to construct a standard multi-sensor ground truth system. Specifically, according to the projection matrix of the camera and the lidar in the ground truth system, the point-by-point three-dimensional coordinates (x, y, z) of the point cloud data are mapped to the pixel coordinates (u, v) on the image data, and the corresponding RGB information on the image data is obtained through the pixel coordinate index, so as to map the image RGB information to the point cloud space to obtain paired RGB-XYZ information, which is used as a supervision signal for training the three-dimensional reconstruction model.

[0076] B2. Image data is obtained by the vehicle-mounted camera, and point cloud data is obtained by the lidar.

[0077] Specifically, the image data output by the camera where H and W represent height and width respectively, each pixel contains three color channels of RGB. The point cloud data output by the laser radar where N represents the number of point clouds, each point p∈P contains three-dimensional coordinates (x, y, z) and reflection intensity i.

[0078] B3, input the image data into the three-dimensional reconstruction model as the key and value of the Transformer, query using the point cloud data, predict the RGB information of the corresponding XYZ coordinates, and obtain the point-by-point RGB prediction result.

[0079] In some embodiments, step B3 specifically comprises:

[0080] B31, feature extraction is performed on the input image data using an image encoder to obtain image features;

[0081] B32, feature extraction is performed on the input point cloud data using a point cloud encoder to obtain point-by-point features;

[0082] B33, the point-by-point features are taken as the query of the cross-attention module, the image features are aggregated, and the point-by-point features are updated;

[0083] B34, the updated point-by-point features are input into a reconstruction prediction head to generate point-by-point RGB prediction results, and the reconstruction of the three-dimensional scene is realized.

[0084] B4, calculate the reconstruction loss according to the aligned RGB-XYZ ground truth information and the point-by-point RGB prediction result, and train the three-dimensional reconstruction model according to the loss.

[0085] The updated point cloud features are input into the reconstruction prediction head to generate point cloud RGB prediction results The reconstruction of the three-dimensional scene is realized. According to the aligned RGB-XYZ supervision signal O rgb and the reconstruction loss function SmoothL1, the reconstruction loss is calculated for model training.

[0086] The trained three-dimensional reconstruction model takes non-calibrated vehicle-mounted sensor information as input and can reconstruct a three-dimensional scene in the ground truth system coordinate system. The trained three-dimensional reconstruction model has the ability of implicit spatial alignment of multi-sensor information, does not rely on the calibration information of multi-sensor, realizes the decoupling of calibration information and model input, takes non-calibrated vehicle-mounted sensor information as input, and can reconstruct a three-dimensional scene in the ground truth system coordinate system, thereby avoiding calibration for each vehicle.

[0087] In summary, the method of the embodiment differs from the existing method in that it explicitly relies on calibration information, decouples the calibration information from the model input, realizes implicit spatial alignment of multi-sensor information, and improves the robustness to calibration noise. In the actual application of the autonomous driving scene, the autonomous driving vehicle constructs a standard ground truth system and provides calibrated RGB-XYZ information. The sensor information equipped on the vehicle is used as input, and the method enables the model to directly reconstruct spatial information consistent with the ground truth system from the sensor information on the vehicle, thereby avoiding calibration for each vehicle.

[0088] Embodiment 2

[0089] The embodiment provides a multi-sensor fusion three-dimensional reconstruction device, comprising:

[0090] A data acquisition module is configured to acquire image data and point cloud data through a sensor.

[0091] A data processing module is configured to input the obtained image data and point cloud data into a trained three-dimensional reconstruction model to generate a point-by-point RGB prediction result, thereby realizing reconstruction of a three-dimensional scene.

[0092] The three-dimensional reconstruction model works as follows:

[0093] Feature extraction is performed on the input image data to obtain image features.

[0094] Feature extraction is performed on the input point cloud data to obtain point-by-point features.

[0095] The point-by-point features are used as queries for a cross-attention module, the image features are aggregated, and the point-by-point features are updated.

[0096] The updated point-by-point features are input into a reconstruction prediction head to generate a point-by-point RGB prediction result.

[0097] Since the device is a multi-sensor fusion three-dimensional reconstruction device of the embodiment, and the principle of solving the problem is similar to the method, the implementation of the device can be referred to the implementation process of the method embodiment, and the repeated parts will not be described again.

[0098] Embodiment 3

[0099] The embodiment of the application also provides an electronic device, which comprises a processor and a memory. The memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a multi-sensor fusion three-dimensional reconstruction method as shown in Figure 1

[0100] ​It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing various method embodiments described above, etc.; and the data storage area can store data created according to the use of the server, etc.

[0101] The processor can include one or more processing cores. The processor connects various parts within the entire server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with a combination of one or more of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs, etc.; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be implemented by a separate chip.

[0102] Since the electronic device is an electronic device corresponding to the multi-sensor fusion three-dimensional reconstruction method of the embodiments of the present application, and the principle of solving problems of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0103] Embodiment 4

[0104] The embodiments of the present application also provide a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement a multi-sensor fusion three-dimensional reconstruction method as shown in Figure 1

[0105] ​Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0106] Since the storage medium is a storage medium corresponding to the multi-sensor fusion three-dimensional reconstruction method of the embodiments of the present application, and the problem solving principle of the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0107] Embodiment 5

[0108] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the multi-sensor fusion three-dimensional reconstruction method according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Wherein, the executable computer program codes or "codes" for executing various embodiments can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0109] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so forth.

[0110] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0111] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A multi-sensor fusion three-dimensional reconstruction method, characterized in that, The method comprises the following steps: obtaining image data and point cloud data through a sensor; inputting the obtained image data and point cloud data into a trained three-dimensional reconstruction model to generate a point-by-point RGB prediction result, so as to realize reconstruction of a three-dimensional scene; wherein the three-dimensional reconstruction model works as follows: extracting features from the input image data to obtain image features; extracting features from the input point cloud data to obtain point-by-point features; using the point-by-point features as queries of a cross-attention module to aggregate features of the image features and update the point-by-point features; inputting the updated point-by-point features into a reconstruction prediction head to generate a point-by-point RGB prediction result; the three-dimensional reconstruction model is a three-dimensional reconstruction model based on a Transformer, and the three-dimensional reconstruction model is trained in the following manner: a set of standard multi-sensor ground truth systems are arranged on a vehicle to provide aligned RGB-XYZ ground truth information; obtaining image data through a first sensor and obtaining point cloud data through a second sensor; constructing a three-dimensional reconstruction model, inputting the image data into the three-dimensional reconstruction model as keys and values of a Transformer, using the point cloud data for querying, predicting RGB information of corresponding XYZ coordinates, and obtaining a point-by-point RGB prediction result; calculating a reconstruction loss according to the RGB-XYZ ground truth information and the point-by-point RGB prediction result, and training the three-dimensional reconstruction model according to the loss; the set of standard multi-sensor ground truth systems arranged on the vehicle to provide aligned RGB-XYZ ground truth information comprises: mapping the point-wise three-dimensional coordinates of the point cloud data to pixel coordinates on the image data according to the projection matrix of the camera and the lidar in the ground truth system ;​ obtaining corresponding RGB information on the image data through pixel coordinate indexing, mapping the RGB information of the image to a point cloud space to obtain RGB-XYZ ground truth information, and using the RGB-XYZ ground truth information for supervised training of the three-dimensional reconstruction model.

2. The multi-sensor fusion three-dimensional reconstruction method of claim 1, wherein, the three-dimensional reconstruction model comprises an image encoder, a point cloud encoder, a cross-attention module, and a reconstruction prediction head; the image encoder is used for extracting features from the input image data to obtain image features; the point cloud encoder is used for extracting features from the input point cloud data to obtain point-by-point features; the cross-attention module is used for aggregating features of the image features and updating the point-by-point features; the reconstruction prediction head is used for generating a point-by-point RGB prediction result according to the updated point-by-point features, so as to realize reconstruction of a three-dimensional scene.

3. The multi-sensor fusion three-dimensional reconstruction method of claim 2, wherein, inputting the image features into the cross-attention module as keys and values, and using the point cloud features to dynamically query the image features; wherein the updated point cloud features contain geometric information and texture information; the calculation formula of the reconstruction loss is as follows: input the updated point cloud features into a reconstruction prediction head to generate a per-point RGB prediction result .

4. The multi-sensor fusion three-dimensional reconstruction method of claim 1, wherein, the multi-sensor fusion three-dimensional reconstruction method further comprises the following steps: wherein, denotes the reconstruction loss function, and denote the point-wise RGB prediction and the point-wise RGB label, respectively.

5. The multi-sensor fusion three-dimensional reconstruction method of claim 1, wherein, after the training of the three-dimensional reconstruction model is completed, the ground truth system is deleted. comprises:

6. A multi-sensor fusion three-dimensional reconstruction apparatus, characterized by, a data acquisition module for obtaining image data and point cloud data through a sensor; ​ The data processing module is configured to input the obtained image data and point cloud data into a trained three-dimensional reconstruction model to generate a point-wise RGB prediction result, so as to realize reconstruction of a three-dimensional scene. The three-dimensional reconstruction model works as follows: Feature extraction is performed on the input image data to obtain image features; Feature extraction is performed on the input point cloud data to obtain point-wise features; The point-wise features are taken as queries of a cross-attention module, and the image features are aggregated and updated based on the point-wise features; The updated point-wise features are input into a reconstruction prediction head to generate a point-wise RGB prediction result; The three-dimensional reconstruction model is a three-dimensional reconstruction model based on a Transformer, and the three-dimensional reconstruction model is trained in the following manner: A standard multi-sensor ground truth system is constructed on a vehicle to provide aligned RGB-XYZ ground truth information; Image data is obtained by a first sensor, and point cloud data is obtained by a second sensor; A three-dimensional reconstruction model is constructed, the image data is input into the three-dimensional reconstruction model as keys and values of a Transformer, the point cloud data is used for querying, and RGB information of corresponding XYZ coordinates is predicted to obtain a point-wise RGB prediction result; A reconstruction loss is calculated based on the RGB-XYZ ground truth information and the point-wise RGB prediction result, and the three-dimensional reconstruction model is trained based on the loss. The standard multi-sensor ground truth system constructed on the vehicle to provide aligned RGB-XYZ ground truth information includes: mapping the point-wise three-dimensional coordinates of the point cloud data to pixel coordinates on the image data according to the projection matrix of the camera and the lidar in the ground truth system ;​ The corresponding RGB information on the image data is obtained through pixel coordinate indexing, the RGB information of the image is mapped to the point cloud space to obtain the RGB-XYZ ground truth information, and the RGB-XYZ ground truth information is used for supervised training of the three-dimensional reconstruction model.

7. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the multi-sensor fusion three-dimensional reconstruction method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the multi-sensor fusion three-dimensional reconstruction method of any one of claims 1 to 5.

Images