Multimodal sensor fusion perception method, computer device, medium and vehicle

Abstract

The present application relates to the technical field of automatic driving, and particularly provides a multi-modal sensor fusion perception method, computer equipment, medium and vehicle, aiming at solving the problem of improving the accuracy of multi-sensor fusion perception. To this end, the method provided by the present application comprises obtaining sensor data obtained by each sensor of a plurality of sensors of different modalities, and using a preset perception model to perform fusion perception on the sensor data obtained by each sensor; wherein, first sensor data obtained by each sensor of a plurality of sensors of different modalities can be obtained; a sensor of one modality is selected as a master sensor, noise is added to the time stamp of the first sensor data obtained by the master sensor to form second sensor data; and the second sensor data and the first sensor data obtained by a non-master sensor are used to train the preset perception model. Through the above method, the robustness and accuracy of multi-modal sensor data fusion perception can be improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a multimodal sensor fusion perception method, computer equipment, medium, and vehicle. Background Technology

[0002] When controlling a vehicle for autonomous driving, sensor data from various modal sensors (such as cameras and LiDAR) on the vehicle can be acquired. Perception models are used to fuse these sensor data to obtain information such as obstacles around the vehicle. Based on this information, the vehicle's driving trajectory can be planned and the vehicle can be controlled to drive autonomously according to the driving trajectory.

[0003] Because the exposure times or scanning times of sensor data obtained from different sensors vary, when fusing sensor data using a perception model, the timestamps of the sensor data from each sensor are first time-aligned before fusion sensing. The time alignment method primarily involves nearest-neighbor matching based on the timestamps of the sensor data. However, in practical applications, there may be significant deviations in the exposure times or scanning times of sensor data obtained from different sensors. In such cases, the timestamps of the sensor data matched together by nearest neighbor may have large discrepancies, leading to significant differences in the measured objects corresponding to these sensor data. If fusion sensing is performed on this sensor data, accurate perception results may not be obtained, thus affecting the safety and reliability of autonomous driving based on the fused perception results.

[0004] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention

[0005] To overcome the above-mentioned defects, this invention is proposed to provide a multimodal sensor fusion sensing method, computer equipment, medium, and vehicle that solves, or at least partially solves, the technical problem of how to eliminate or reduce the impact of large timestamp deviations between different sensor data on multimodal sensor fusion sensing, so as to improve the accuracy of multimodal sensor fusion sensing.

[0006] In a first aspect, a multimodal sensor fusion sensing method is provided, the method comprising:

[0007] Acquire sensor data from each sensor in multiple sensors with different modes;

[0008] A preset perception model is used to fuse and perceive the sensor data obtained from each sensor.

[0009] The preset perception model is trained in the following way:

[0010] Acquire the first sensor data obtained by each sensor in multiple sensors with different modes;

[0011] A sensor of one mode is selected as the main sensor, and noise is added to the timestamp of the first sensor data obtained by the main sensor to form the second sensor data;

[0012] The preset perception model is trained using the data from the second sensor and the data from the first sensor obtained from a non-master sensor.

[0013] In one technical solution of the above-mentioned multimodal sensor fusion sensing method, the step of "training the preset sensing model using the second sensor data and the first sensor data obtained from a non-master sensor" specifically includes:

[0014] From the first sensor data obtained from the non-master sensor, obtain the first sensor data whose timestamp is closest to that of the second sensor data;

[0015] The preset perception model is trained using the second sensor data and the first sensor data whose timestamp is closest to it.

[0016] In one technical solution of the aforementioned multimodal sensor fusion sensing method,

[0017] The step of “acquiring the first sensor data obtained by each sensor in multiple sensors of different modes” specifically includes: acquiring the time sequence of sensor data obtained by each sensor, wherein the time sequence of sensor data includes multiple single-frame first sensor data based on time sequence arrangement;

[0018] The step of “adding noise to the timestamp of the first sensor data obtained by the main sensor” specifically includes: adding the same noise or different noise to the timestamp of each single frame of the first sensor data in the time sequence of the sensor data, so as to form each single frame of the second sensor data in the time sequence of the sensor data.

[0019] In one technical solution of the above-mentioned multimodal sensor fusion sensing method, the method further includes obtaining the first sensor data whose timestamp is closest to the second sensor data by means of the following method, while adding the same noise:

[0020] For each single frame of second sensor data in the time sequence of sensor data from the main sensor, from each single frame of first sensor data in the time sequence of sensor data from the non-main sensor,

[0021] Obtain the nearest neighbor of the first sensor data in a single frame that precedes the timestamp of the second sensor data in this single frame;

[0022] or,

[0023] For each single frame of second sensor data in the time sequence of sensor data from the main sensor, from each single frame of first sensor data in the time sequence of sensor data from the non-main sensor,

[0024] Obtain the nearest neighbor of the first sensor data in a single frame after the timestamp of the second sensor data in this single frame.

[0025] In one technical solution of the above-mentioned multimodal sensor fusion sensing method, the method further includes obtaining the first sensor data whose timestamp is closest to the second sensor data by means of adding different noise:

[0026] For each single frame of second sensor data in the time sequence of sensor data from the main sensor, from each single frame of first sensor data in the time sequence of sensor data from the non-main sensor,

[0027] Randomly obtain the nearest neighbor of the first sensor data in a single frame that is before or after the timestamp of the second sensor data in this single frame.

[0028] In one technical solution of the above-mentioned multimodal sensor fusion sensing method, the step of "adding noise to the timestamp of the first sensor data obtained by the main sensor" specifically includes:

[0029] Obtain the timestamp difference between two adjacent single-frame first sensor data obtained from a non-primary sensor;

[0030] Based on the timestamp difference, noise is added to the timestamp of the first sensor data obtained by the main sensor;

[0031] Wherein, the noise is less than the timestamp difference, and the frequency at which the non-master sensor acquires sensor data is greater than the frequency at which the master sensor acquires sensor data.

[0032] In one technical solution of the above-mentioned multimodal sensor fusion sensing method, the step of "selecting a sensor of one modality as the main sensor" specifically includes:

[0033] The accuracy of sensor data obtained from sensors in each mode;

[0034] The sensor with the highest accuracy mode is selected as the main sensor.

[0035] In a second aspect, a computer device is provided, comprising a processor and a storage device, the storage device being adapted to store a plurality of program codes, the program codes being adapted to be loaded and run by the processor to perform the method described in any of the above-described technical solutions of the multimodal sensor fusion sensing method.

[0036] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the method described in any of the above-described technical solutions of the multimodal sensor fusion sensing method.

[0037] In a fourth aspect, a vehicle is provided that includes the computer equipment described in the above-described computer equipment technical solution.

[0038] The above-described technical solutions of the present invention have at least one or more of the following beneficial effects:

[0039] In implementing the multimodal sensor fusion perception method provided by this invention, sensor data obtained from each sensor in multiple different modalities can be acquired, and a preset perception model is used to fuse the sensor data obtained from each sensor. The preset perception model is trained as follows: acquiring first sensor data from each sensor in multiple different modalities; selecting one modality as the master sensor and adding noise to the timestamp of the first sensor data obtained by the master sensor to form second sensor data; and training the preset perception model using the second sensor data and the first sensor data obtained by the non-master sensor. By training the perception model in this way, it can accurately complete fusion perception even when there are large deviations in the exposure time or data scanning time of the sensor data obtained from different sensors. This improves the robustness and accuracy of multimodal sensor data fusion perception using this perception model, thereby enhancing the safety and reliability of autonomous driving using the fusion perception results. Attached Figure Description

[0040] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Wherein:

[0041] Figure 1 This is a schematic flowchart of the main steps of a multimodal sensor fusion sensing method according to an embodiment of the present invention;

[0042] Figure 2 This is a schematic flowchart of the main steps of a method for obtaining a perception model according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic flowchart of the main steps of a method for training a perception model using data from the first and second sensors according to an embodiment of the present invention.

[0044] Figure 4 This is a schematic diagram illustrating the acquisition of first sensor data that is nearest to the timestamp of a single frame of second sensor data according to an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram illustrating the acquisition of the first sensor data that is nearest to the timestamp of the time sequence of sensor data according to an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the main structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0047] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0048] In the description of this invention, "processor" can include hardware, software, or a combination of both. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc.

[0049] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.

[0050] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.

[0051] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

[0052] The following describes an embodiment of the fusion sensing method for multimodal sensors.

[0053] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a multimodal sensor fusion sensing method according to an embodiment of the present invention. Figure 1 As shown, the multimodal sensor fusion sensing method in this embodiment of the invention mainly includes the following steps S101 to S102.

[0054] Step S101: Acquire sensor data from each sensor in multiple sensors with different modalities. Step S102: Perform fusion sensing on the sensor data obtained from each sensor using a preset sensing model.

[0055] Sensor modes can be categorized based on sensor type; for example, the sensor type can be used as a mode. Taking a camera as an example, its type is an image sensor, and its mode can also be an image sensor.

[0056] Taking autonomous driving as an example, multiple sensors of different modalities can include cameras and LiDAR. The sensor data obtained by the camera is an image, and the sensor data obtained by the LiDAR is a point cloud. When controlling the autonomous driving of a vehicle, the cameras and LiDAR on the vehicle can be used to collect images and point clouds around the vehicle, respectively. Then, a preset perception model is used to fuse these images and point clouds for perception. The result of the fusion perception can include lane line information, obstacle information, etc. around the vehicle. Based on the result of the fusion perception, the vehicle's driving path is planned and the vehicle is controlled to drive along the driving path.

[0057] The following explains the method for obtaining the preset perception model. For example... Figure 2 As shown, in this embodiment of the invention, the perception model can be trained through the following steps S201 to S203.

[0058] Step S201: Acquire the first sensor data obtained by each sensor among multiple sensors of different modes. The first sensor data can be the raw sensor data obtained by the sensor. In contrast, the second sensor data obtained through the subsequent step S202 is no longer the raw sensor data.

[0059] Model training typically involves multiple iterations until a preset convergence condition is met, at which point training stops. This preset convergence condition can be either the model's training performance metrics (such as accuracy) meeting a predefined threshold, or the number of training iterations reaching a predefined threshold.

[0060] For each training iteration, training can be performed using a single frame of sensor data or using multi-frame time-series sensor data. For these two training methods, different approaches can be used to acquire the first sensor data from each sensor.

[0061] 1. Training using single-frame sensor data

[0062] In this scenario, the first frame of sensor data obtained from each sensor can be acquired separately, and then combined into a single sensor data sample. This sensor data sample is used for training in each iteration. For example, if different modal sensors include a camera and a LiDAR, then the single-frame image (i.e., one frame image) acquired by the camera and the single-frame point cloud (i.e., one frame point cloud) acquired by the LiDAR can be combined into a single sample.

[0063] 2. Training using time-series multi-frame sensor data

[0064] In this case, the time series sequences of sensor data obtained from each sensor can be obtained separately, and the time series sequences of sensor data obtained from each sensor can be combined into a sensor data sample. This sensor data sample is used for training in each iteration of training.

[0065] The sensor data time series includes multiple single-frame first sensor data arranged in time sequence. Taking a camera as an example, the obtained sensor data time series can be three frames of images arranged from first to last according to the acquisition time.

[0066] It should be noted that although the above two training methods can be used to train the perception model in the embodiments of the present invention, the training method used in each iteration of training needs to be consistent.

[0067] Step S202: Select a sensor of one mode as the main sensor, add noise to the timestamp of the first sensor data obtained by the main sensor to form the second sensor data.

[0068] Those skilled in the art can flexibly select a sensor of one modality as the main sensor according to actual needs. In some preferred embodiments, the main sensor can be selected from multiple modal sensors through the following steps S2021 to S2022.

[0069] Step S2021: Obtain the accuracy of the sensor data obtained from each mode of the sensor. Step S2022: Select the sensor with the highest accuracy as the main sensor.

[0070] The higher the accuracy of sensor data, the more accurately the sensor data describes or represents the information of the measured object. For example, in a scenario where different modal sensors include cameras and lidar, and the measured object is a lane line, the camera acquires a lane line image, while the lidar acquires a lane line point cloud. Since the lane line point cloud provides a better description or representation of the lane line position than the lane line image, the lidar can be selected as the main sensor. This embodiment of the invention does not specifically limit the method for obtaining the accuracy of the sensor data obtained by each sensor, as long as the accuracy of the sensor data obtained by each sensor can be obtained.

[0071] In some implementations, noise can be added to the timestamp of the first sensor data obtained by the main sensor through the following steps S2023 to S2024.

[0072] Step S2023: Obtain the timestamp difference between two adjacent single-frame first sensor data obtained from the non-master sensor.

[0073] Each frame of sensor data acquired by the sensor carries a timestamp, which indicates the moment the sensor data was collected. Since the time interval between any two adjacent frames of sensor data is basically the same, two adjacent frames of first sensor data can be arbitrarily selected, and the difference between the timestamps of these two frames can be calculated.

[0074] Step S2024: Add noise to the timestamps of the first sensor data obtained by the main sensor based on the timestamp difference. The added noise is smaller than the timestamp difference, and the frequency at which non-main sensors acquire sensor data is greater than the frequency at which the main sensor acquires sensor data. For example, the frequency at which the camera acquires images is greater than the frequency at which the LiDAR acquires point clouds.

[0075] Noise is time information with a duration. When adding noise to the timestamp of the first sensor data, the duration corresponding to the noise can be increased or decreased.

[0076] As described in step S201 above, training can be performed using single-frame sensor data or time-series multi-frame sensor data. Different methods can be used to add noise to these two methods.

[0077] 1. Training using single-frame sensor data

[0078] In this case, noise can be randomly added to the timestamp of each single frame of the first sensor data. By randomly adding noise, the time interval between any two single frames of the second sensor data formed after adding noise can also be randomized, thereby improving the diversity of the second sensor data.

[0079] 2. Training using time-series multi-frame sensor data

[0080] In this scenario, the same or different noise can be added to the timestamps of each frame of the first sensor data in the sensor data time series to form each frame of the second sensor data in the sensor data time series. By adding the same noise, it can be ensured that the time interval between any two adjacent frames of the second sensor data remains essentially the same. By adding different noise, the time interval between any two adjacent frames of the second sensor data can be different, thus improving the diversity of the second sensor data.

[0081] Step S203: Use the data from the second sensor and the data from the first sensor obtained from the non-master sensor to train a preset perception model.

[0082] In this embodiment of the invention, conventional model training methods can be used to train the perception model using data from the second sensor and data from the first sensor obtained from a non-primary sensor, which will not be elaborated here.

[0083] Based on the method described in steps S201 to S203 above, the sensing model can accurately complete fusion sensing even when the exposure time or data scanning time of sensor data obtained from different sensors deviates significantly, thereby improving the robustness and accuracy of multimodal sensor data fusion sensing based on the method described in steps S101 to S102 above.

[0084] For example, in one application scenario of the multimodal sensor fusion perception method according to the present invention, different modal sensors include a camera and a lidar. The perception model is trained using images obtained from the camera and point clouds obtained from the lidar, enabling the trained perception model to perceive vehicle information in the environment based on the images and point clouds. Under normal circumstances, due to the different image acquisition frequencies of the camera and the point cloud scanning frequencies of the lidar, the images and point clouds obtained by the camera and lidar for the same measured object will differ by approximately 20ms. In practical applications, due to malfunctions of the camera or lidar, the images and point clouds obtained by the two for the same measured object may differ by 40ms. Since 40ms is greater than 20ms, the point cloud describes or represents all or most of the information of the measured object, while the image only describes or represents a small portion of the information. If the perception model matches the point cloud and the image together for fusion perception, the perception result may be incorrect due to significant interference in the image. For example, a stone pier might be perceived as a vehicle. To address this, the multimodal sensor fusion sensing method provided by this invention can be used to add noise (e.g., reduce by 60ms) to the timestamps of the point cloud. This allows the trained sensing model to have better anti-interference capability against the image when matching the point cloud with the image for fusion sensing, and ultimately still obtain a relatively accurate sensing result.

[0085] The following provides a further explanation of step S203.

[0086] In some embodiments of step S203 above, it can be achieved by... Figure 3 The following steps S2031 to S2032 are shown to train a preset perception model.

[0087] Step S2031: From the first sensor data obtained from the non-master sensor, obtain the first sensor data whose timestamp is closest to that of the second sensor data.

[0088] Specifically, the difference between the timestamp of the first sensor data obtained by the non-master sensor and the timestamp of the second sensor data can be obtained, and then the first sensor data with the smallest difference can be selected as the first sensor data with the nearest timestamp.

[0089] Step S2032: Using the second sensor data and the first sensor data whose timestamp is closest to it, a preset perception model is trained.

[0090] Specifically, the second sensor data and the first sensor data whose timestamp is closest to it can be combined into a sensor data sample, and such a sensor data sample can be used for training in each iteration.

[0091] Based on the method described in steps S2031 to S2032 above, time alignment of the second sensor data and the first sensor data obtained from the non-master sensor is achieved, thereby improving the training effect of training the perception model using the second sensor data and the first sensor data obtained from the non-master sensor, and improving the perception capability of the perception model.

[0092] The following provides a further explanation of step S2031.

[0093] As described in step S201 above, training can be performed using single-frame sensor data or time-series multi-frame sensor data. For these two methods, different methods can be used to obtain the first sensor data whose timestamp is closest to the second sensor data, which will be explained below.

[0094] (a) Training using single-frame sensor data

[0095] As can be seen from the aforementioned step S2024, when adding noise to the timestamp of the first sensor data, the time length corresponding to the noise can be randomly increased or decreased for this timestamp.

[0096] If the time length is increased, then when acquiring the first sensor data whose timestamp is closest to the second sensor data, the first sensor data whose timestamp is closest to the second sensor data can be acquired from the first sensor data whose timestamp is located after the timestamp of the second sensor data.

[0097] If the time length is reduced, the first sensor data whose timestamp is the nearest neighbor can be obtained from the first sensor data that is located before the timestamp of this second sensor data.

[0098] like Figure 4 As shown, the first sensor data is the image obtained by the camera, and the second sensor data is the point cloud obtained by the LiDAR. If no noise is added to the timestamps of the images, then the image whose timestamp is closest to the point cloud's is... Figure 4 The image frame connected to the point cloud by a solid line. If noise is added to the image's timestamp, then the image whose timestamp is closest to the point cloud's might be... Figure 4 One of the two images connected to the dotted line of the point cloud.

[0099] (ii) Training using time-series multi-frame sensor data

[0100] As described in step S202 above, in this case, the same noise or different noise can be added to the timestamps of the first sensor data in each single frame of the sensor data time sequence. Different methods can be used to obtain the first sensor data with the nearest timestamp for adding the same or different noise, which will be explained below.

[0101] 1. Add the same noise

[0102] In some embodiments of step S2031 above, for each single frame of second sensor data in the time sequence of sensor data of the main sensor, the nearest neighbor of the first sensor data in a single frame that is before the timestamp of the second sensor data in a single frame can be obtained from each single frame of first sensor data in the time sequence of sensor data of the non-main sensor.

[0103] like Figure 5 As shown, the first sensor data is the image obtained by the camera, and the second sensor data is the point cloud obtained by the lidar. Figure 5 The diagram shows an image time series consisting of multiple frames and a point cloud time series consisting of three point cloud frames with timestamps T-4, T-2, and T. Without adding noise to the image timestamps, the image closest to the timestamps of these three point cloud frames is the frame connected to the point cloud by the solid line. With noise added to the image timestamps, the image closest to the timestamps of these three point cloud frames is... Figure 5 One of the two image frames connected to the point cloud by the dashed line. In this step, the image frame connected by the dashed line to the left of the solid line can be obtained as the first sensor data frame with the nearest time stamp.

[0104] In some embodiments of step S2031 above, for each single frame of second sensor data in the time sequence of sensor data of the main sensor, the nearest neighbor of the single frame of first sensor data located after the timestamp of the single frame of second sensor data can be obtained from each single frame of first sensor data in the time sequence of sensor data of the non-main sensor.

[0105] Please refer to the appendix for further details. Figure 5 In this step, the frame image connected by the dashed line to the right of the solid line can be obtained as the first sensor data of the single frame with the nearest time stamp.

[0106] 2. Add different noises

[0107] In some embodiments of step S2031 above, for each single frame of second sensor data in the time sequence of sensor data of the main sensor, the nearest neighboring single frame of first sensor data located before or after the timestamp of that single frame of second sensor data can be randomly obtained from each single frame of first sensor data in the time sequence of sensor data of the non-main sensor. (Continue to refer to the appendix...) Figure 5 In this step, the frame image connected by the dashed line to the left or right of the solid line can be obtained as the first sensor data of the single frame with the nearest time stamp.

[0108] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in the present invention and therefore will also fall within the protection scope of the present invention.

[0109] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0110] Furthermore, the present invention also provides a computer device.

[0111] See appendix Figure 6 , Figure 6 This is a schematic diagram of the main structure of a computer device according to an embodiment of the present invention. Figure 6 As shown, the computer device in this embodiment of the invention mainly includes a storage device and a processor. The storage device can be configured to store a program for executing the multimodal sensor fusion sensing method of the above-described method embodiments. The processor can be configured to execute the program in the storage device, which includes, but is not limited to, a program for executing the multimodal sensor fusion sensing method of the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of the present invention.

[0112] In embodiments of the present invention, the computer device may be a control device comprising various electronic devices. In some possible implementations, the computer device may include multiple storage devices and multiple processors. The program executing the multimodal sensor fusion sensing method of the above method embodiments can be divided into multiple subroutines, each subroutine can be loaded and run by a processor to execute different steps of the multimodal sensor fusion sensing method of the above method embodiments. Specifically, each subroutine can be stored in different storage devices, and each processor can be configured to execute programs in one or more storage devices to jointly implement the multimodal sensor fusion sensing method of the above method embodiments, that is, each processor executes different steps of the multimodal sensor fusion sensing method of the above method embodiments to jointly implement the multimodal sensor fusion sensing method of the above method embodiments.

[0113] The aforementioned multiple processors can be processors deployed on the same device. For example, the aforementioned computer device can be a high-performance device composed of multiple processors, and the aforementioned multiple processors can be processors configured on that high-performance device. Alternatively, the aforementioned multiple processors can also be processors deployed on different devices. For example, the aforementioned computer device can be a server cluster, and the aforementioned multiple processors can be processors on different servers within the server cluster.

[0114] Furthermore, the present invention also provides a computer-readable storage medium.

[0115] In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium may be configured to store a program for executing the multimodal sensor fusion sensing method of the above-described method embodiments. This program may be loaded and run by a processor to implement the multimodal sensor fusion sensing method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium may be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0116] Furthermore, the present invention also provides a vehicle.

[0117] In one embodiment of a vehicle according to the present invention, the vehicle may include the computer equipment described in the above-described computer equipment embodiments. In this embodiment, the vehicle may be an autonomous vehicle, an unmanned vehicle, or the like. Furthermore, according to the type of power source, the vehicle in this embodiment may be a gasoline vehicle, an electric vehicle, a hybrid vehicle that combines electric and gasoline power, or a vehicle using other new energy sources, etc.

[0118] The technical solution of the present invention has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A multimodal sensor fusion sensing method, characterized in that, The method includes: Acquire sensor data from each sensor in multiple different modalities, the sensor data including images; A preset perception model is used to fuse and perceive the sensor data obtained from each sensor. The preset perception model is trained in the following way: Acquire the first sensor data obtained by each sensor in multiple sensors with different modes; A sensor of one mode is selected as the main sensor, and noise is added to the timestamp of the first sensor data obtained by the main sensor to form the second sensor data; The preset perception model is trained using the data from the second sensor and the data from the first sensor obtained from a non-primary sensor. Adding noise to the timestamps of the first sensor data obtained by the main sensor includes: obtaining the timestamp difference between two adjacent single frames of first sensor data obtained by a non-main sensor; adding noise to the timestamps of the first sensor data obtained by the main sensor based on the timestamp difference; wherein the noise is less than the timestamp difference, and the frequency at which the non-main sensor obtains sensor data is greater than the frequency at which the main sensor obtains sensor data.

2. The method according to claim 1, characterized in that, The step of "training the preset perception model using the second sensor data and the first sensor data obtained from the non-primary sensor" specifically includes: From the first sensor data obtained from the non-master sensor, obtain the first sensor data whose timestamp is closest to that of the second sensor data; The preset perception model is trained using the second sensor data and the first sensor data whose timestamp is closest to it.

3. The method according to claim 1 or 2, characterized in that, The step of "acquiring the first sensor data obtained by each sensor in multiple sensors of different modes" specifically includes: acquiring the time sequence of sensor data obtained by each sensor, wherein the time sequence of sensor data includes multiple single-frame first sensor data based on time sequence arrangement; The step of "adding noise to the timestamp of the first sensor data obtained by the main sensor" specifically includes: adding the same noise or different noise to the timestamp of each single frame of the first sensor data in the time sequence of the sensor data, so as to form each single frame of the second sensor data in the time sequence of the sensor data.

4. The method according to claim 3, characterized in that, The method further includes obtaining first sensor data that is the nearest neighbor of the timestamp of the second sensor data by means of the following method, while adding the same noise: For each single frame of second sensor data in the time sequence of sensor data from the main sensor, from each single frame of first sensor data in the time sequence of sensor data from the non-main sensor, Obtain the nearest neighbor of the first sensor data in a single frame that precedes the timestamp of the second sensor data in this single frame; or, For each single frame of second sensor data in the time sequence of sensor data from the main sensor, from each single frame of first sensor data in the time sequence of sensor data from the non-main sensor, Obtain the nearest neighbor of the first sensor data in a single frame after the timestamp of the second sensor data in this single frame.

5. The method according to claim 3, characterized in that, The method further includes obtaining first sensor data that is the nearest neighbor of the timestamp of the second sensor data by means of adding different noise: For each single frame of second sensor data in the time sequence of sensor data from the main sensor, from each single frame of first sensor data in the time sequence of sensor data from the non-main sensor, Randomly obtain the nearest neighbor of the first sensor data in a single frame that is before or after the timestamp of the second sensor data in this single frame.

6. The method according to claim 1, characterized in that, The steps of "selecting a sensor of a certain mode as the main sensor" specifically include: The accuracy of sensor data obtained from sensors in each mode; The sensor with the highest accuracy mode is selected as the main sensor.

7. A computer device comprising a processor and a storage device, said storage device being adapted to store a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by the processor to perform the fusion sensing method of the multimodal sensor according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the fusion sensing method of the multimodal sensor according to any one of claims 1 to 6.

9. A vehicle, characterized in that, The vehicle includes the computer equipment as described in claim 7.

Images