Multi-sensor observation state prediction model training method and device, automatic driving vehicle fusion positioning method and device, electronic equipment and computer program product

Abstract

The invention discloses a training method and device of a multi-sensor observation state prediction model, a fusion positioning method and device of an automatic driving vehicle, electronic equipment and a computer program product. The training method comprises the following steps: determining whether the automatic driving vehicle meets an online training condition at present; if yes, constructing training sample data of the observation state prediction model, including RTK positioning data and observation data of other sensors; marking the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain observation state marking information; and training an observation state prediction model by using observation data and observation state labeling information of other sensors. According to the method, the online training of the observation state prediction model can be realized through the judgment of the online training condition, the method does not depend on high-precision truth value equipment, is more flexible and lower in cost, the trained model provides more powerful support for the selection of multiple observation information sources, and the model is easy to deploy and maintain.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a training method and device for a multi-sensor observation state prediction model, a fusion positioning method and device for an autonomous driving vehicle, an electronic device, and a computer program product. Background Art

[0002] In the field of autonomous driving, fusion positioning technology is becoming increasingly important as a key link in achieving autonomous navigation and precise control of vehicles. This technology aims to improve the accuracy and reliability of positioning by integrating observation information from different sensors, such as radar, LiDAR, cameras, and global positioning systems (GPS), thereby ensuring that autonomous vehicles can operate safely and efficiently in complex and changing road environments.

[0003] Traditionally, the autonomous driving fusion positioning solution mainly relies on algorithm logic and voting mechanism to determine which sensor information source is the most reliable and should be used first. Although this solution has achieved the initial fusion of multi-sensor information to a certain extent, its selection strategy may face severe challenges in certain critical situations. Specifically, when the information provided by multiple sensors is contradictory or at a fuzzy boundary, the traditional voting mechanism often finds it difficult to make accurate judgments, which affects the accuracy of the positioning results and may endanger the overall performance and safety of the autonomous driving system.

[0004] In addition, with the continuous advancement of autonomous driving technology and the increasing complexity of road environments, the requirements for fusion positioning technology are also increasing. The limitations and shortcomings of traditional judgment methods based on algorithm logic and voting are becoming increasingly obvious when dealing with positioning needs in complex scenarios and extreme conditions. Therefore, how to develop a more intelligent, accurate and adaptable autonomous driving fusion positioning technology to overcome the problems of traditional solutions and improve positioning accuracy and reliability has become an important issue that needs to be urgently addressed in the current field of autonomous driving technology. Summary of the invention

[0005] The embodiments of the present application provide a training method and device for a multi-sensor observation state prediction model, a fusion positioning method and device for an autonomous driving vehicle, an electronic device, and a computer program product to achieve online training of the observation state prediction model and improve the accuracy of the observation state prediction.

[0006] The present application embodiment adopts the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a method for training a multi-sensor observation state prediction model, wherein the multi-sensor is deployed on an autonomous driving vehicle, and the method for training the multi-sensor observation state prediction model includes:

[0008] Determine whether the autonomous vehicle currently meets the online training conditions of the observation state prediction model;

[0009] When the autonomous vehicle currently meets the online training conditions of the observation state prediction model, construct the training sample data of the observation state prediction model, where the training sample data includes RTK positioning data and observation data of other sensors;

[0010] Annotate the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain the observation state annotation information;

[0011] Use the observation data of other sensors and the corresponding observation state annotation information to train the observation state prediction model to obtain the trained observation state prediction model.

[0012] Optionally, the determining whether the autonomous vehicle currently meets the online training conditions of the observation state prediction model includes:

[0013] Obtain the RTK positioning data, the RTK positioning state, and the observation data of other sensors;

[0014] If the RTK positioning state is the differential state, determine the positioning error between the RTK positioning data and the observation data of other sensors according to the RTK positioning data and the observation data of other sensors;

[0015] If the positioning error between the RTK positioning data and the observation data of at least one other sensor is less than the preset error threshold, determine that the autonomous vehicle currently meets the online training conditions of the observation state prediction model;

[0016] If the RTK positioning state is the non-differential state, or the positioning error between the RTK positioning data and the observation data of all other sensors is not less than the preset error threshold, determine that the autonomous vehicle currently does not meet the online training conditions of the observation state prediction model.

[0017] Optionally, the annotating the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain the observation state annotation information includes:

[0018] Calculate the error between the observation data of each other sensor and the RTK positioning data respectively;

[0019] Determine the observation state identifier corresponding to the observation data of other sensors according to the error between the observation data of each other sensor and the RTK positioning data.

[0020] Optionally, the training of the observation state prediction model using the observation data of other sensors and the corresponding observation state annotation information to obtain the trained observation state prediction model includes:

[0021] Obtain the confidence corresponding to the observation data of other sensors;

[0022] Use the observation data of other sensors, the corresponding confidence, and the observation state annotation information to train the observation state prediction model to obtain the trained observation state prediction model.

[0023] In a second aspect, an embodiment of the present application further provides a fusion positioning method for an autonomous driving vehicle, where the fusion positioning method for the autonomous driving vehicle includes:

[0024] Determine whether the autonomous driving vehicle currently meets the observation state prediction condition;

[0025] When the autonomous driving vehicle currently meets the observation state prediction condition, obtain the observation data of other sensors;

[0026] Input the observation data of other sensors into the observation state prediction model to obtain the observation state prediction result of other sensors;

[0027] Perform fusion positioning based on the observation data of other sensors and the observation state prediction result of other sensors to obtain the fusion positioning result of the autonomous driving vehicle;

[0028] Wherein, the observation state prediction model is trained based on the training method of the observation state prediction model of the multi-sensor described in any one of the foregoing.

[0029] Optionally, the performing fusion positioning based on the observation data of other sensors and the observation state prediction result of other sensors to obtain the fusion positioning result of the autonomous driving vehicle includes:

[0030] Obtain the observation state identifier output by the original observation state algorithm;

[0031] Determine the target observation data based on the observation state identifier output by the original observation state algorithm and the observation state prediction result;

[0032] Perform fusion positioning based on the target observation data to obtain the fusion positioning result of the autonomous driving vehicle.

[0033] Optionally, the determining the target observation data based on the observation state identifier output by the original observation state algorithm and the observation state prediction result includes:

[0034] Determine whether the observation state identifier output by the original observation state algorithm is consistent with the observation state prediction result;

[0035] If they are consistent, directly use the observation data of the sensor corresponding to the observation status identifier or the observation status prediction result as the target observation data;

[0036] If they are inconsistent, fuse the observation data of the sensor corresponding to the observation status identifier and the observation data of the sensor corresponding to the observation status prediction result to obtain the fused observation data and use it as the target observation data.

[0037] In a third aspect, an embodiment of the present application further provides a training device for an observation status prediction model of multiple sensors. The training device for the observation status prediction model of multiple sensors includes:

[0038] A first determination unit for determining whether an autonomous driving vehicle currently meets the online training conditions of the observation status prediction model;

[0039] A construction unit for constructing training sample data of the observation status prediction model when the autonomous driving vehicle currently meets the online training conditions of the observation status prediction model. The training sample data includes RTK positioning data and observation data of other sensors;

[0040] A labeling unit for labeling the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain observation status labeling information;

[0041] A training unit for training the observation status prediction model by using the observation data of other sensors and the corresponding observation status labeling information to obtain a trained observation status prediction model.

[0042] In a fourth aspect, an embodiment of the present application further provides a fusion positioning device for an autonomous driving vehicle. The fusion positioning device for the autonomous driving vehicle includes:

[0043] A second determination unit for determining whether an autonomous driving vehicle currently meets the observation status prediction conditions;

[0044] An acquisition unit for acquiring the observation data of other sensors when the autonomous driving vehicle currently meets the observation status prediction conditions;

[0045] A prediction unit for inputting the observation data of other sensors into the observation status prediction model to obtain the observation status prediction results of other sensors;

[0046] A fusion positioning unit for performing fusion positioning according to the observation data of other sensors and the observation status prediction results of other sensors to obtain the fusion positioning result of the autonomous driving vehicle;

[0047] Among them, the observation state prediction model is obtained by training based on the training device of the observation state prediction model of the foregoing multi-sensors.

[0048] In a fifth aspect, an embodiment of the present application further provides an electronic device, including:

[0049] a processor; and a memory arranged to store computer-executable instructions, where the executable instructions, when executed, cause the processor to execute any one of the foregoing training methods for the observation state prediction model of the multi-sensors, or execute any one of the foregoing fusion positioning methods for the autonomous driving vehicle.

[0050] In a sixth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instruction, where the computer program or instruction is executed by the processor to execute any one of the foregoing training methods for the observation state prediction model of the multi-sensors, or execute any one of the foregoing fusion positioning methods for the autonomous driving vehicle.

[0051] At least one of the foregoing technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: The training method for the observation state prediction model of the multi-sensors in the embodiments of the present application, where the multi-sensors are deployed on an autonomous driving vehicle, and the training method for the observation state prediction model of the multi-sensors includes: first determining whether the autonomous driving vehicle currently meets the online training condition of the observation state prediction model; then, when the autonomous driving vehicle currently meets the online training condition of the observation state prediction model, constructing training sample data for the observation state prediction model, where the training sample data includes RTK positioning data and observation data of other sensors; then annotating the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain observation state annotation information; and finally training the observation state prediction model using the observation data of other sensors and the corresponding observation state annotation information to obtain a trained observation state prediction model. The training method for the observation state prediction model of the multi-sensors in the embodiments of the present application can realize the online training of the observation state prediction model through the judgment of the online training condition. The model training does not depend on high-precision true value devices, is more flexible and has lower costs. The trained observation state prediction model can provide stronger support for the subsequent selection of multi-sensor observation information sources, and the model is easy to deploy and maintain. Compared with the traditional logic voting method, the accuracy and robustness are higher. Description of the Drawings

[0052] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0053] Figure 1It is a schematic flowchart of a training method for an observation state prediction model of multiple sensors in an embodiment of the present application;

[0054] Figure 2 It is a schematic flowchart of a fusion positioning method for an autonomous driving vehicle in an embodiment of the present application;

[0055] Figure 3 It is a schematic structural diagram of a training device for an observation state prediction model of multiple sensors in an embodiment of the present application;

[0056] Figure 4 It is a schematic structural diagram of a fusion positioning device for an autonomous driving vehicle in an embodiment of the present application;

[0057] Figure 5 It is a schematic structural diagram of an electronic device in an embodiment of the present application. Detailed implementation manners

[0058] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Apparently, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0059] The following will, with reference to the drawings, elaborate on the technical solutions provided in each embodiment of the present application.

[0060] The embodiment of the present application provides a training method for an observation state prediction model of multiple sensors. As Figure 1 shown, a schematic flowchart of a training method for an observation state prediction model of multiple sensors in an embodiment of the present application is provided. The multiple sensors are deployed on an autonomous driving vehicle. The training method for the observation state prediction model of the multiple sensors at least includes the following steps S110 to S140:

[0061] Step S110, determine whether the autonomous driving vehicle currently meets the online training conditions of the observation state prediction model.

[0062] The multi-sensors in the embodiments of the present application refer to various sensor devices or modules deployed on an autonomous vehicle, such as an RTK positioning device, a lidar, a camera, an IMU (Inertial Measurement Unit), etc. When training the observation state prediction model, it is necessary to first determine whether the autonomous vehicle currently meets the conditions for online training. Because in the online training scenario, there is generally no high-precision ground truth device on the actually operating autonomous vehicle, and it is necessary to be able to determine sufficiently accurate positioning data from other sensors as the ground truth data. Therefore, the embodiments of the present application set the online training conditions for the purpose of collecting the ground truth data.

[0063] Step S120, when the autonomous vehicle currently meets the online training conditions of the observation state prediction model, construct the training sample data of the observation state prediction model, where the training sample data includes RTK positioning data and the observation data of other sensors.

[0064] When the autonomous vehicle currently meets the online training conditions of the observation state prediction model, construct the training sample data of the observation state prediction model. The training sample data in the embodiments of the present application is mainly composed of RTK positioning data and the observation data output by other sensors except the RTK positioning device.

[0065] Considering the actual driving scenario of the autonomous vehicle and the positioning accuracy and stability of different sensors, the embodiments of the present application select the RTK positioning device that is generally deployed on the autonomous vehicle to replace the high-precision ground truth device, and use the RTK positioning data output by the RTK positioning device when the online training conditions are met as the ground truth data.

[0066] The observation data of other multi-sensors mainly includes point cloud data collected based on the lidar and lidar SLAM positioning data (Lidar-SLAM) output by the SLAM algorithm, image data collected based on the camera and visual SLAM positioning data (Val-SLAM) output by the SLAM algorithm, visual positioning data (Val) output based on the image data of the camera and the traditional visual algorithm, positioning data (DR) obtained by dead reckoning based on the IMU data, etc. Of course, specifically which types of sensor observation data are included can be flexibly determined by those skilled in the art according to the actual situation.

[0067] Step S130, label the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain the observation state labeling information.

[0068] The RTK positioning data obtained when the online training condition is satisfied can be regarded as data with sufficiently accurate positioning results. Therefore, it can be used as a basis for evaluating the positioning errors of other multi-sensor systems, and then the observation states of each group of multi-sensor observation data can be automatically labeled according to the magnitudes of the multi-sensor positioning errors. A group of multi-sensor observation data can be understood as the observation data corresponding to the multi-sensors at the same moment after time synchronization processing. The labeled observation state identifier represents the identifier corresponding to the sensor with the highest observation information accuracy in each group of multi-sensor observation data.

[0069] Step S140: Use the observation data of other sensors and the corresponding observation state annotation information to train the observation state prediction model, and obtain the trained observation state prediction model.

[0070] The observation state annotation information corresponding to the observation data of other sensors can be used as a supervision signal for model training. A neural network is used to train the observation data of other sensors so that the trained model can accurately predict the observation state identifier corresponding to the selected sensor. The specific neural network used can be, for example, a BP neural network (feedforward neural network), which mainly consists of an input layer, a hidden layer, and an output layer. Of course, it can also be other types of neural networks, which are not specifically limited here.

[0071] The training method of the multi-sensor observation state prediction model in the embodiments of the present application can realize the online training of the observation state prediction model through the judgment of the online training condition. The model training does not rely on high-precision true value devices, does not require a large amount of data to be collected offline, is more flexible and has lower costs. The trained observation state prediction model can provide stronger support for the subsequent selection of multi-sensor observation information sources, and the model is easy to deploy and maintain. Compared with the traditional logic voting method, it has higher accuracy and robustness.

[0072] In some embodiments of the present application, determining whether the autonomous vehicle currently meets the online training condition of the observation state prediction model includes: obtaining RTK positioning data, RTK positioning status, and the observation data of other sensors; if the RTK positioning status is a differential status, then determine the positioning error between the RTK positioning data and the observation data of other sensors according to the RTK positioning data and the observation data of other sensors; if the positioning error between the RTK positioning data and the observation data of at least one other sensor is less than a preset error threshold, then determine that the autonomous vehicle currently meets the online training condition of the observation state prediction model; if the RTK positioning status is a non-differential status, or the positioning error between the RTK positioning data and the observation data of all other sensors is not less than the preset error threshold, then determine that the autonomous vehicle currently does not meet the online training condition of the observation state prediction model.

[0073] For the purpose of realizing online training, it is necessary to be able to collect RTK positioning data with high enough accuracy as true value data. Therefore, the judgment on whether the online training conditions are met mainly depends on whether the RTK positioning accuracy meets the accuracy requirements for true value data.

[0074] Specifically, the embodiments of the present application can make a judgment based on the RTK positioning status output by the RTK positioning device. If the RTK positioning status is a differential status, in order to avoid the "spoofing" behavior of the RTK positioning device, the observation data of other sensors can be further combined to assist in judging the accuracy of the RTK positioning data. For example, if the positioning error between the RTK positioning data and the observation data of at least one other sensor is less than a preset error threshold, it is considered that the RTK positioning data meets the accuracy requirements. Only at this time is it considered that the online training conditions are met, that is, the acquisition conditions for training sample data are met, and the RTK positioning data and the observation data of other sensors are synchronously collected to form training sample data.

[0075] If the RTK positioning status is a non-differential status, or the positioning error between the RTK positioning data and the observation data of all other sensors is not less than the preset error threshold, it is considered that the current RTK positioning data has low accuracy and does not meet the acquisition conditions for training sample data.

[0076] Through the judgment of the above online training conditions, the sample data volume that meets the model training requirements can be gradually accumulated, and new data can also be continuously accumulated for iterative update of the model in the future.

[0077] In some embodiments of the present application, the labeling of the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain observation status labeling information includes: calculating the error between the observation data of each other sensor and the RTK positioning data respectively; determining the observation status identifier corresponding to the observation data of other sensors according to the error between the observation data of each other sensor and the RTK positioning data.

[0078] When labeling the observation data of other sensors, the positioning error magnitude between the observation data of each other sensor and the true value data can be calculated respectively in the dimension of the observation data of each group of multi-sensors. The larger the positioning error, the lower the observation accuracy of the sensor, and the smaller the positioning error, the higher the observation accuracy of the sensor. In the actual observation selection scenario, the observation data of the sensor with the highest observation accuracy should be selected as the main observation information source. Therefore, the embodiments of the present application can determine the sensor with the smallest positioning error from the positioning error magnitudes between the observation data of each other sensor and the true value data of the RTK positioning, and mark the observation status identifier of this group of data with the label corresponding to the sensor with the smallest positioning error.

[0079] Specifically, the observation status identifiers of multiple sensors can be set in advance, such as represented as Lidar - SLAM - 2, Val - SLAM - 3, Val - 4, DR - 5. If the error between the observation data output by Lidar - SLAM and the true value data is the smallest in a set of observation data of multiple sensors, then the observation status identifier of this set of data is marked as Lidar - SLAM - 2.

[0080] Through the above method, automatic annotation of sample data can be achieved, which greatly improves the annotation efficiency and reduces the annotation cost compared with the manual annotation method.

[0081] In some embodiments of the present application, training the observation status prediction model by using the observation data of other sensors and the corresponding observation status annotation information to obtain the trained observation status prediction model includes: obtaining the confidence level corresponding to the observation data of other sensors; using the observation data of other sensors, the corresponding confidence level, and the observation status annotation information to train the observation status prediction model to obtain the trained observation status prediction model.

[0082] Considering that each sensor outputs the confidence level information corresponding to the observation data while outputting the observation data. For example, laser SLAM can output the confidence score of the laser SLAM positioning data. The higher the confidence score, the higher the accuracy of the laser SLAM positioning data. Visual SLAM can output the confidence score of the visual SLAM positioning data. The higher the confidence score, the higher the accuracy of the visual SLAM positioning data.

[0083] Based on this, in the embodiments of the present application, when training the observation status prediction model, the confidence level information output by multiple other sensors can also be incorporated into the model training process as prior knowledge, thereby improving the convergence speed of the model.

[0084] The embodiments of the present application also provide a fusion positioning method for an autonomous driving vehicle, as Figure 2 shown, which provides a schematic flowchart of a fusion positioning method for an autonomous driving vehicle in the embodiments of the present application. The fusion positioning method for the autonomous driving vehicle at least includes the following steps S210 to step S240:

[0085] Step S210, determining whether the autonomous driving vehicle currently meets the observation status prediction condition.

[0086] When performing integrated positioning of an autonomous vehicle using the observation state prediction model of multiple sensors trained based on the foregoing embodiments, it is necessary to first determine whether the autonomous vehicle currently meets the observation state prediction conditions. Here, it mainly involves determining the positioning state of the RTK positioning device on the autonomous vehicle. If the RTK positioning state is the differential state, it indicates that the current RTK positioning data meets the positioning accuracy requirements. Generally, there is no need to switch the sensor observation information source, and thus there is no need to predict the observation state of multiple sensors. Of course, to ensure the accuracy of the determination, the positioning error between the observation data of other sensors and the RTK positioning data can also be further combined to assist in verifying the RTK positioning state. Specifically, reference can be made to the determination of the online training conditions in the foregoing embodiments. If the RTK positioning state is the non-differential state, it indicates that the RTK positioning accuracy is not high, and it is necessary to select the current main observation information source from other sensors, that is, the observation state prediction conditions are met at this time.

[0087] Step S220, when the autonomous vehicle currently meets the observation state prediction conditions, obtain the observation data of other sensors.

[0088] If the autonomous vehicle currently meets the observation state prediction conditions, obtain the observation data of other sensors. For example, it may include laser SLAM positioning data, visual SLAM positioning data, visual positioning data, positioning data obtained by dead reckoning, etc.

[0089] Step S230, input the observation data of other sensors into the observation state prediction model to obtain the observation state prediction results of other sensors.

[0090] Input the observation data collected by the above multiple sensors into the trained observation state prediction model of multiple sensors. The observation state prediction model of multiple sensors can directly output the observation state prediction results of multiple sensors. Specifically, it can output an observation state identifier. For example, if the output observation state identifier is Lidar-SLAM-2, it indicates that in the current observation state, laser SLAM positioning should be selected or switched as the main observation information source.

[0091] Step S240, perform integrated positioning based on the observation data of other sensors and the observation state prediction results of other sensors to obtain the integrated positioning result of the autonomous vehicle; wherein, the observation state prediction model is trained based on the training method of the observation state prediction model of multiple sensors described in any one of the foregoing.

[0092] After determining the current main information source, the fusion weights corresponding to the observation information sources of each sensor can be adaptively adjusted according to the determined main information source, that is, increasing the weight of the main information source and reducing the weights of other information sources. Then, the observation data of other sensors and the corresponding fusion weights are input into the Kalman filter for fusion positioning, so as to obtain the fusion positioning result of the autonomous driving vehicle.

[0093] Based on the pre-trained observation state prediction model of multiple sensors, the embodiments of the present application can accurately predict the observation state identifiers of multiple sensors in real time, providing more accurate and reliable support for the selection of multi-sensor observation information sources, especially for the selection of multi-sensor observation information sources in special road scenarios. Compared with the existing method based on algorithm logic voting judgment, it has higher robustness and accuracy, avoiding the problem of inaccurate selection in critical situations. The fusion positioning is carried out based on the prediction results of the multi-sensor observation states, improving the positioning accuracy and stability of the autonomous driving vehicle in some special scenarios.

[0094] In some embodiments of the present application, the fusion positioning according to the observation data of other sensors and the prediction results of the observation states of other sensors to obtain the fusion positioning result of the autonomous driving vehicle includes: obtaining the observation state identifier output by the original observation state algorithm; determining the target observation data according to the observation state identifier output by the original observation state algorithm and the observation state prediction result; and performing fusion positioning according to the target observation data to obtain the fusion positioning result of the autonomous driving vehicle.

[0095] In practical applications, the embodiments of the present application do not abandon the original method of voting and selecting the observation information source based on algorithm logic. The main principle of the original algorithm logic is to calculate the positioning error and variance between the positioning data of two sensors respectively, and vote to select the best observation information source by combining the positioning error and variance between the positioning data of two sensors.

[0096] Based on the original algorithm and the multi-sensor observation state prediction model of the foregoing embodiments, the embodiments of the present application can respectively obtain a result of selecting the observation information source. In order to take into account the effects of these two methods, the results obtained by the two methods can be compared, and the target observation data finally used for fusion positioning can be determined according to the comparison result.

[0097] In some embodiments of the present application, determining the target observation data based on the observation state identifier output by the original observation state algorithm and the observation state prediction result includes: determining whether the observation state identifier output by the original observation state algorithm is consistent with the observation state prediction result; if they are consistent, directly using the observation data of the sensor corresponding to the observation state identifier or the observation state prediction result as the target observation data; if they are inconsistent, fusing the observation data of the sensor corresponding to the observation state identifier and the observation data of the sensor corresponding to the observation state prediction result to obtain the fused observation data and using it as the target observation data.

[0098] If the observation information sources selected by the two methods are the same, then the observation data of this observation information source can be directly used as the target observation data for subsequent fusion positioning. If the observation information sources selected by the two methods are different, the results of the two methods can be comprehensively considered. For example, if the original algorithm selects visual SLAM positioning and the model predicts laser SLAM positioning, then the visual SLAM positioning and the laser SLAM positioning can be weighted and fused for observation information to obtain the target observation data. Of course, the specific size of the weight can be flexibly set and adjusted according to the actual situation and is not specifically limited here.

[0099] The embodiment of the present application also provides a training device 300 for an observation state prediction model of multiple sensors, as Figure 3 shown, which provides a structural schematic diagram of a training device for an observation state prediction model of multiple sensors in the embodiment of the present application. The training device 300 for an observation state prediction model of multiple sensors includes: a first determination unit 310, a construction unit 320, a labeling unit 330, and a training unit 340, where:

[0100] The first determination unit 310 is configured to determine whether the autonomous driving vehicle currently meets the online training conditions of the observation state prediction model;

[0101] The construction unit 320 is configured to construct training sample data of the observation state prediction model when the autonomous driving vehicle currently meets the online training conditions of the observation state prediction model. The training sample data includes RTK positioning data and observation data of other sensors;

[0102] The labeling unit 330 is configured to label the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain observation state labeling information;

[0103] The training unit 340 is configured to train the observation state prediction model by using the observation data of other sensors and the corresponding observation state labeling information to obtain a trained observation state prediction model.

[0104] In some embodiments of the present application, the first determination unit 310 is specifically configured to: obtain RTK positioning data, RTK positioning status, and observation data of other sensors; if the RTK positioning status is a differential status, determine the positioning error between the RTK positioning data and the observation data of other sensors according to the RTK positioning data and the observation data of other sensors; if the positioning error between the RTK positioning data and the observation data of at least one other sensor is less than a preset error threshold, determine that the autonomous vehicle currently meets the online training condition of the observation state prediction model; if the RTK positioning status is a non-differential status, or the positioning error between the RTK positioning data and the observation data of all other sensors is not less than the preset error threshold, determine that the autonomous vehicle currently does not meet the online training condition of the observation state prediction model.

[0105] In some embodiments of the present application, the annotation unit 330 is specifically configured to: calculate the error between the observation data of each other sensor and the RTK positioning data respectively; determine the observation state identifier corresponding to the observation data of other sensors according to the error between the observation data of each other sensor and the RTK positioning data.

[0106] In some embodiments of the present application, the training unit 340 is specifically configured to: obtain the confidence corresponding to the observation data of other sensors; train the observation state prediction model by using the observation data of other sensors, the corresponding confidence, and the observation state annotation information to obtain a trained observation state prediction model.

[0107] It can be understood that the above training device for the observation state prediction model of multiple sensors can implement each step of the training method for the observation state prediction model of multiple sensors provided in the foregoing embodiments. The relevant explanations for the training method for the observation state prediction model of multiple sensors are applicable to the training device for the observation state prediction model of multiple sensors, and will not be elaborated here.

[0108] An embodiment of the present application further provides a fusion positioning device 400 for an autonomous vehicle, as Figure 4 shown, which provides a schematic structural diagram of a fusion positioning device for an autonomous vehicle in an embodiment of the present application. The fusion positioning device 400 for an autonomous vehicle includes: a second determination unit 410, an acquisition unit 420, a prediction unit 430, and a fusion positioning unit 440, where:

[0109] The second determination unit 410 is configured to determine whether the autonomous vehicle currently meets the observation state prediction condition;

[0110] The acquisition unit 420 is configured to obtain the observation data of other sensors when the autonomous vehicle currently meets the observation state prediction condition;

[0111] A prediction unit 430 is configured to input the observation data of other sensors into an observation state prediction model to obtain an observation state prediction result of the other sensors;

[0112] A fusion positioning unit 440 is configured to perform fusion positioning based on the observation data of other sensors and the observation state prediction result of the other sensors to obtain a fusion positioning result of the autonomous driving vehicle;

[0113] Wherein, the observation state prediction model is trained based on the training device of the observation state prediction model of the foregoing multi-sensors.

[0114] In some embodiments of the present application, the fusion positioning unit 440 is specifically configured to: obtain an observation state identifier output by an original observation state algorithm; determine target observation data according to the observation state identifier output by the original observation state algorithm and the observation state prediction result; perform fusion positioning based on the target observation data to obtain a fusion positioning result of the autonomous driving vehicle.

[0115] In some embodiments of the present application, the fusion positioning unit 440 is specifically configured to: determine whether the observation state identifier output by the original observation state algorithm is consistent with the observation state prediction result; if they are consistent, directly use the observation data of the sensor corresponding to the observation state identifier or the observation state prediction result as the target observation data; if they are inconsistent, fuse the observation data of the sensor corresponding to the observation state identifier and the observation data of the sensor corresponding to the observation state prediction result to obtain fused observation data and use it as the target observation data.

[0116] It can be understood that the above-mentioned fusion positioning device of the autonomous driving vehicle can implement each step of the fusion positioning method of the autonomous driving vehicle provided in the foregoing embodiments. The relevant explanations about the fusion positioning method of the autonomous driving vehicle are applicable to the fusion positioning device of the autonomous driving vehicle, and will not be elaborated here.

[0117] Figure 5 It is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 5 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0118] The processor, network interface, and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, Figure 5 only a bidirectional arrow is used in Figure 5 , but it does not mean that there is only one bus or one type of bus.

[0119] Memory, which is used to store programs. Specifically, the program can include program code, and the program code includes computer operation instructions. The memory can include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0120] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a training device for the observation state prediction model of multiple sensors or a fusion positioning device for an autonomous driving vehicle at the logical level. The processor executes the program stored in the memory.

[0121] The above-mentioned method executed by the training device for the observation state prediction model of multiple sensors disclosed in the embodiments as shown in this application Figure 1 or Figure 2The method executed by the fusion positioning device of the autonomous driving vehicle disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed by the integrated logic circuit of the hardware in the processor or instructions in the form of software. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0122] The embodiments of the present application also propose a computer program product. The computer program product stores one or more programs, and the one or more programs include instructions that, when executed by an electronic device including a plurality of application programs, can enable the electronic device to execute Figure 1 the method executed by the training device of the observation state prediction model of multiple sensors in the illustrated embodiment, or execute Figure 2 the method executed by the fusion positioning device of the autonomous driving vehicle in the illustrated embodiment.

[0123] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0124] The present invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each flow and / or block of the flowchart illustrations and / or block diagrams, and combinations of flows and / or blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart flow or flows and / or block or blocks. Figure 1 in a flow or flows and / or block or blocks Figure 1 or blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the functions specified in the flowchart flow or flows and / or block or blocks. Figure 1 in a flow or flows and / or block or blocks Figure 1 or blocks.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and / or block or blocks. Figure 1 in a flow or flows and / or block or blocks Figure 1 or blocks.

[0127] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0128] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory. The memory is an example of computer-readable media.

[0129] A computer-readable medium includes permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage, or other magnetic storage devices, or any other non-transitory media that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0130] It should also be noted that the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a series of elements includes not only those elements but also other elements not expressly listed, or elements that are inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0131] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, system, or computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0132] The above are only the embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A training method for a multi-sensor observation state prediction model, wherein: Multiple sensors are deployed on an autonomous driving vehicle, and a training method for an observation state prediction model of the multiple sensors includes: Determine whether the autonomous driving vehicle currently meets the online training conditions of the observation state prediction model; When the autonomous driving vehicle currently meets the online training conditions of the observation state prediction model, constructing training sample data of the observation state prediction model, wherein the training sample data includes RTK positioning data and observation data of other sensors; Annotating the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain observation status annotation information; The observation data of other sensors and the corresponding observation state annotation information are used to train the observation state prediction model to obtain a trained observation state prediction model.

2. The method for training a multi-sensor observation state prediction model according to claim 1, wherein: The step of determining whether the autonomous driving vehicle currently meets the online training conditions of the observation state prediction model includes: Obtain RTK positioning data and RTK positioning status as well as observation data from other sensors; If the RTK positioning state is a differential state, determining a positioning error between the RTK positioning data and the observation data of other sensors according to the RTK positioning data and the observation data of other sensors; If the positioning error between the RTK positioning data and the observation data of at least one other sensor is less than a preset error threshold, it is determined that the autonomous driving vehicle currently meets the online training conditions of the observation state prediction model; If the RTK positioning state is a non-differential state, or the positioning error between the RTK positioning data and the observation data of all other sensors is not less than a preset error threshold, it is determined that the autonomous driving vehicle currently does not meet the online training conditions of the observation state prediction model.

3. The method for training a multi-sensor observation state prediction model according to claim 1, wherein: The step of labeling the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain the observation status labeling information includes: Calculating the error between the observation data of each other sensor and the RTK positioning data respectively; The observation state identifier corresponding to the observation data of other sensors is determined according to the error between the observation data of each other sensor and the RTK positioning data.

4. The method for training a multi-sensor observation state prediction model according to claim 1, wherein: The method of using the observation data of other sensors and the corresponding observation state annotation information to train the observation state prediction model to obtain the trained observation state prediction model includes: Obtain the confidence level corresponding to the observation data of other sensors; The observation data of other sensors, the corresponding confidence levels and the observation state annotation information are used to train the observation state prediction model to obtain a trained observation state prediction model.

5. A fusion positioning method for an autonomous driving vehicle, wherein: The fusion positioning method of the autonomous driving vehicle includes: Determine whether the autonomous driving vehicle currently meets the observation state prediction conditions; When the autonomous driving vehicle currently meets the observation state prediction conditions, the observation data of other sensors are obtained; Inputting the observation data of other sensors into the observation state prediction model to obtain the observation state prediction results of other sensors; Based on the observation data of other sensors and the prediction results of the observation states of other sensors, fusion positioning is performed to obtain the fusion positioning result of the autonomous driving vehicle; Wherein, the observation state prediction model is trained based on the training method of the observation state prediction model of the multi-sensor according to any one of claims 1 to 4.

6. The fusion positioning method for an autonomous driving vehicle according to claim 5, wherein: The fusion positioning of the autonomous driving vehicle is performed based on the observation data of other sensors and the observation state prediction results of other sensors, and the fusion positioning result obtained includes: Get the observation state identifier output by the original observation state algorithm; Determine the target observation data according to the observation state identifier output by the original observation state algorithm and the observation state prediction result; Fusion positioning is performed based on the target observation data to obtain the fusion positioning result of the autonomous driving vehicle.

7. The fusion positioning method for an autonomous driving vehicle according to claim 6, wherein: The determining of the target observation data according to the observation state identifier output by the original observation state algorithm and the observation state prediction result comprises: Determining whether the observation state identifier output by the original observation state algorithm is consistent with the observation state prediction result; If they are consistent, the observation data of the sensor corresponding to the observation state identifier or the observation state prediction result is directly used as the target observation data; If they are inconsistent, the observation data of the sensor corresponding to the observation state identifier and the observation data of the sensor corresponding to the observation state prediction result are fused to obtain the fused observation data as the target observation data.

8. A training device for a multi-sensor observation state prediction model, wherein: The training device of the multi-sensor observation state prediction model comprises: A first determination unit, used to determine whether the autonomous driving vehicle currently meets the online training conditions of the observation state prediction model; A construction unit, configured to construct training sample data of the observation state prediction model when the autonomous driving vehicle currently meets the online training conditions of the observation state prediction model, wherein the training sample data includes RTK positioning data and observation data of other sensors; A labeling unit, used for labeling the observation data of other sensors according to the RTK positioning data and the observation data of other sensors to obtain observation state labeling information; The training unit is used to train the observation state prediction model using the observation data of other sensors and the corresponding observation state annotation information to obtain a trained observation state prediction model.

9. A fusion positioning device for an autonomous driving vehicle, wherein: The fusion positioning device of the autonomous driving vehicle includes: A second determination unit, used to determine whether the autonomous driving vehicle currently meets the observation state prediction condition; An acquisition unit, used to acquire observation data of other sensors when the autonomous driving vehicle currently meets the observation state prediction condition; A prediction unit, used to input the observation data of other sensors into the observation state prediction model to obtain the observation state prediction results of other sensors; A fusion positioning unit is used to perform fusion positioning based on the observation data of other sensors and the observation state prediction results of other sensors to obtain a fusion positioning result of the autonomous driving vehicle; Wherein, the observation state prediction model is trained based on the training device of the observation state prediction model of the multi-sensor according to claim 8.

10. An electronic device comprising: processor; And a memory arranged to store computer executable instructions, which, when executed, cause the processor to execute the training method of the multi-sensor observation state prediction model of any one of claims 1 to 4, or execute the fusion positioning method of the autonomous driving vehicle of any one of claims 5 to 7.

11. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the training method of the multi-sensor observation state prediction model described in any one of claims 1 to 4, or executes the fusion positioning method of the autonomous driving vehicle described in any one of claims 5 to 7.

Images