Multi-sensor observation state prediction model training method and device, multi-sensor observation state prediction method and device, electronic equipment and computer program product

Abstract

The invention discloses a multi-sensor observation state prediction model training method and device, a multi-sensor observation state prediction method and device, electronic equipment and a computer program product, and the training method comprises the steps: constructing training sample data of an observation state prediction model, the training sample data comprises observation data of multiple sensors and truth value data of high-precision truth value equipment; labeling the observation data according to the observation data and the truth value data of the multiple sensors to obtain observation state labeling information; and training an observation state prediction model by using the observation data of the multiple sensors and the corresponding observation state labeling information. According to the invention, on the basis of truth value data provided by high-precision truth value equipment, automatic labeling of multi-sensor observation data is realized; in combination with observation data and observation state labeling information of multiple sensors, the neural network is adopted to train the observation state prediction model, powerful support is provided for subsequent selection of observation information sources of the multiple sensors, and the model is easy to deploy and maintain.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular, to a training method and device for an observation state prediction model of multi-sensors, an observation state prediction method and device of multi-sensors, an electronic device, and a computer program product. Background Art

[0002] In the field of autonomous driving, the fusion positioning technology, as a key link to achieve vehicle autonomous navigation and precise control, has become increasingly important. This technology aims to improve the positioning accuracy and reliability by integrating observation information from different sensors, such as radar, Light Detection and Ranging (LiDAR), cameras, and Global Positioning System (GPS), etc., so as to ensure that autonomous driving vehicles can operate safely and efficiently in complex and changeable road environments.

[0003] Traditionally, autonomous driving fusion positioning solutions mainly rely on algorithm logic and voting mechanisms to determine which current sensor information source is the most reliable and should be preferentially adopted. Although this solution has achieved the preliminary fusion of multi-sensor information to a certain extent, in some critical situations, its selection strategy may face severe challenges. Specifically, when the information provided by multiple sensors is contradictory or at a fuzzy boundary, the traditional voting mechanism often has difficulty making accurate judgments, resulting in the accuracy of the positioning result being affected, and thus may endanger the overall performance and safety of the autonomous driving system.

[0004] In addition, with the continuous progress of autonomous driving technology and the increasing complexity of road environments, the requirements for fusion positioning technology are also constantly improving. The limitations and deficiencies of traditional judgment methods based on algorithm logic and voting are becoming more and more obvious when dealing with positioning requirements in complex scenarios and extreme conditions. Therefore, how to develop a more intelligent, accurate, and adaptable autonomous driving fusion positioning technology to overcome the problems existing in traditional solutions and improve the accuracy and reliability of positioning has become an urgent problem to be solved in the current field of autonomous driving technology. Summary of the Invention

[0005] Embodiments of this application provide a training method and device for an observation state prediction model of multi-sensors, an observation state prediction method and device of multi-sensors, an electronic device, and a computer program product to improve the accuracy of multi-sensor observation state judgment.

[0006] Embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a training method for an observation state prediction model of multi-sensors, where the multi-sensors are deployed on an autonomous driving vehicle, and the training method for the observation state prediction model of the multi-sensors includes:

[0008] Construct training sample data for an observation state prediction model, where the training sample data includes observation data from multiple sensors and true value data from a high-precision true value device;

[0009] Label the observation data of the multiple sensors according to the observation data of the multiple sensors and the true value data of the high-precision true value device to obtain observation state labeling information;

[0010] Train an observation state prediction model using the observation data of the multiple sensors and the corresponding observation state labeling information to obtain a trained observation state prediction model and deploy it to an autonomous vehicle.

[0011] Optionally, the labeling of the observation data of the multiple sensors according to the observation data of the multiple sensors and the true value data of the high-precision true value device to obtain observation state labeling information includes:

[0012] Calculate the error between the observation data of each sensor and the true value data respectively to obtain the observation error of each sensor;

[0013] Determine the observation state identifier corresponding to the observation data of the multiple sensors according to the observation errors of the respective sensors.

[0014] Optionally, the training of the observation state prediction model using the observation data of the multiple sensors and the corresponding observation state labeling information to obtain a trained observation state prediction model and deploy it to an autonomous vehicle includes:

[0015] Obtain the confidence level corresponding to the observation data of the multiple sensors;

[0016] Train an observation state prediction model using the observation data of the multiple sensors, the corresponding confidence level, and the observation state labeling information to obtain a trained observation state prediction model and deploy it to an autonomous vehicle.

[0017] Optionally, the construction of the training sample data for the observation state prediction model includes:

[0018] Obtain the observation data of the multiple sensors of the autonomous vehicle in the target scenario and the true value data of the high-precision true value device, where the target scenario includes at least one of a tunnel, an urban canyon, and an open road;

[0019] Construct the training sample data for the observation state prediction model according to the observation data of the multiple sensors in the target scenario and the true value data of the high-precision true value device.

[0020] In a second aspect, an embodiment of the present application further provides a method for predicting the observation state of multiple sensors, where the method for predicting the observation state of multiple sensors includes:

[0021] Obtain the observation data of the multiple sensors;

[0022] Input the observation data of multiple sensors into the observation state prediction model to obtain the observation state prediction results of the multiple sensors;

[0023] Among them, the observation state prediction model is trained based on the training method of the observation state prediction model of the multiple sensors described in any one of the foregoing.

[0024] Optionally, after inputting the observation data of the multiple sensors into the observation state prediction model to obtain the observation state prediction results of the multiple sensors, the observation state prediction method of the multiple sensors further includes:

[0025] Dynamically adjust the fusion weights corresponding to the observation information of each sensor according to the observation state prediction results of the multiple sensors;

[0026] Perform fusion positioning according to the observation information of each sensor and the corresponding fusion weights to obtain the fusion positioning results.

[0027] In a third aspect, an embodiment of the present application further provides a training device for an observation state prediction model of multiple sensors. Among them, the training device for the observation state prediction model of the multiple sensors includes:

[0028] A construction unit for constructing training sample data of the observation state prediction model, where the training sample data includes the observation data of multiple sensors and the true value data of a high-precision true value device;

[0029] A labeling unit for labeling the observation data of the multiple sensors according to the observation data of the multiple sensors and the true value data of the high-precision true value device to obtain observation state labeling information;

[0030] A training unit for training the observation state prediction model by using the observation data of the multiple sensors and the corresponding observation state labeling information to obtain a trained observation state prediction model and deploy it to an autonomous driving vehicle.

[0031] In a fourth aspect, an embodiment of the present application further provides an observation state prediction device for multiple sensors. Among them, the observation state prediction device for the multiple sensors includes:

[0032] An acquisition unit for acquiring the observation data of multiple sensors;

[0033] A prediction unit for inputting the observation data of the multiple sensors into the observation state prediction model to obtain the observation state prediction results of the multiple sensors;

[0034] Among them, the observation state prediction model is trained based on the training device for the observation state prediction model of the multiple sensors described above.

[0035] Fifth aspect, an embodiment of the present application further provides an electronic device, including:

[0036] a processor; and a memory arranged to store computer-executable instructions, which when executed cause the processor to execute the training method of the observation state prediction model of any one of the foregoing multi-sensors, or execute the observation state prediction method of any one of the foregoing multi-sensors.

[0037] Sixth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instruction, which when executed by a processor executes the training method of the observation state prediction model of any one of the foregoing multi-sensors, or executes the observation state prediction method of any one of the foregoing multi-sensors.

[0038] The above at least one technical solution adopted in the embodiments of the present application can achieve the following beneficial effects: In the training method of the observation state prediction model of the multi-sensors in the embodiments of the present application, the multi-sensors are deployed on an autonomous driving vehicle. First, training sample data of the observation state prediction model is constructed, and the training sample data includes the observation data of the multi-sensors and the true value data of a high-precision true value device; then, the observation data of the multi-sensors is labeled according to the observation data of the multi-sensors and the true value data of the high-precision true value device to obtain observation state labeling information; finally, the observation state prediction model is trained using the observation data of the multi-sensors and the corresponding observation state labeling information to obtain a trained observation state prediction model and deploy it on the autonomous driving vehicle. The training method of the observation state prediction model of the multi-sensors in the embodiments of the present application is based on the true value data provided by the high-precision true value device to realize the automatic labeling of the observation data of the multi-sensors; combined with the observation data of the multi-sensors and the corresponding observation state labeling information, a neural network is used to train the observation state prediction model of the multi-sensors, which provides strong support for the subsequent selection of multi-sensor observation information sources, and the model is easy to deploy and maintain. Compared with the traditional logic voting method, the accuracy and robustness are higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0040] Figure 1 is a schematic flowchart of a training method of an observation state prediction model of a multi-sensor in an embodiment of the present application;

[0041] Figure 2 is a schematic flowchart of an observation state prediction method of a multi-sensor in an embodiment of the present application;

[0042] Figure 3Schematic diagram of the structure of a training device for an observation state prediction model of multiple sensors in an embodiment of the present application;

[0043] Figure 4 Schematic diagram of the structure of an observation state prediction device for multiple sensors in an embodiment of the present application;

[0044] Figure 5 Schematic diagram of the structure of an electronic device in an embodiment of the present application. Detailed implementation manners

[0045] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0046] The following will, with reference to the drawings, elaborate on the technical solutions provided by each embodiment of the present application.

[0047] An embodiment of the present application provides a training method for an observation state prediction model of multiple sensors. As Figure 1 shown, a flowchart of a training method for an observation state prediction model of multiple sensors in an embodiment of the present application is provided. The multiple sensors are deployed on an autonomous vehicle. The training method for the observation state prediction model of the multiple sensors at least includes the following steps S110 to S130:

[0048] Step S110: Construct training sample data for the observation state prediction model. The training sample data includes the observation data of multiple sensors and the true value data of a high-precision true value device.

[0049] The multiple sensors in the embodiment of the present application refer to various sensor devices or modules deployed on an autonomous vehicle, such as RTK positioning devices, lidar, cameras, IMUs (inertial measurement units), etc. When training the observation state prediction model, it is necessary to first construct the training sample data for the observation state prediction model. The training sample data here mainly includes the observation data of multiple sensors and the true value data of a high-precision true value device.

[0050] The observation data of the multi-sensor can specifically include RTK positioning data (RTK) output by an RTK positioning device, point cloud data collected based on lidar, and lidar SLAM positioning data (Lidar-SLAM) output by a SLAM algorithm, image data collected based on a camera and visual SLAM positioning data (Val-SLAM) output by a SLAM algorithm, visual positioning data (Val) output by image data of a camera and a traditional visual algorithm, positioning data (DR) obtained by dead reckoning based on IMU data, etc. Of course, specifically including which types of sensor observation data can be flexibly determined by those skilled in the art according to the actual situation.

[0051] The high-precision ground truth device is a positioning device composed of a combination of high-precision sensors, and its price is relatively expensive. Therefore, a high-precision ground truth device is generally not configured on an autonomous vehicle in an actual application scenario. In the embodiment of the present application, the high-precision ground truth device is mainly deployed for a specific vehicle in the model training stage to collect the positioning data of the high-precision ground truth device as the ground truth data.

[0052] Step S120: Label the observation data of the multi-sensor according to the observation data of the multi-sensor and the ground truth data of the high-precision ground truth device to obtain observation state label information.

[0053] The ground truth data of the high-precision ground truth device obtained in the foregoing step can be regarded as data with sufficiently accurate positioning results. Therefore, it can be used as a basis for evaluating the positioning error of other multi-sensors, and then the observation state of each group of multi-sensor observation data can be labeled according to the size of the multi-sensor positioning error. A group of multi-sensor observation data can be understood as the observation data corresponding to the multi-sensor at the same moment after time synchronization processing. The labeled observation state identifier represents the identifier corresponding to the sensor with the highest observation information accuracy in each group of multi-sensor observation data.

[0054] Step S130: Train an observation state prediction model using the observation data of the multi-sensor and the corresponding observation state label information, and deploy the trained observation state prediction model to the autonomous vehicle.

[0055] The observation state label information corresponding to the observation data of the multi-sensor can be used as a supervision signal for model training. A neural network is used to train the observation data of the multi-sensor so that the trained model can accurately predict the observation state identifier corresponding to the selected sensor. The specifically used neural network can be, for example, a BP neural network (feedforward neural network), which mainly consists of an input layer, a hidden layer, and an output layer. Of course, it can also be other types of neural networks, which are not specifically limited herein.

[0056] The training method of the observation state prediction model for multi-sensors in the embodiments of the present application is based on the true value data provided by a high-precision true value device to achieve automatic annotation of the observation data of multi-sensors. By combining the observation data of multi-sensors and the corresponding observation state annotation information, a neural network is used to train the observation state prediction model for multi-sensors, which provides strong support for the subsequent judgment and selection of the observation state of multi-sensors, and is easy to deploy and maintain. Compared with the traditional logic voting method, it has higher accuracy and robustness.

[0057] In some embodiments of the present application, the annotation of the observation data of multi-sensors according to the observation data of multi-sensors and the true value data of a high-precision true value device to obtain the observation state annotation information includes: calculating the error between the observation data of each sensor and the true value data respectively to obtain the observation error of each sensor; determining the observation state identifier corresponding to the observation data of multi-sensors according to the observation error of each sensor.

[0058] When annotating the observation data of multi-sensors, the positioning error magnitude between the observation data of each sensor and the true value data can be calculated respectively in the dimension of each group of observation data of multi-sensors. The larger the positioning error, the lower the observation accuracy of the sensor; the smaller the positioning error, the higher the observation accuracy of the sensor. In the actual observation selection scenario, the observation data of the sensor with the highest observation accuracy should be selected as the main observation information source. Therefore, in the embodiments of the present application, the sensor with the smallest positioning error can be determined from the positioning error magnitudes between the observation data of each sensor and the true value data, and the observation state identifier of this group of data is marked as the identifier corresponding to the sensor with the smallest positioning error.

[0059] Specifically, the observation state identifiers of multi-sensors can be set in advance, such as represented as RTK-1, Lidar-SLAM-2, Val-SLAM-3, Val-4, DR-5. If the error between the observation data output by Lidar-SLAM and the true value data is the smallest in a group of observation data of multi-sensors, then the observation state identifier of this group of data is marked as Lidar-SLAM-2.

[0060] Through the above method, automatic annotation of sample data can be achieved, which greatly improves the annotation efficiency and reduces the annotation cost compared with the manual annotation method.

[0061] In some embodiments of the present application, training an observation state prediction model using the observation data of multiple sensors and the corresponding observation state annotation information, obtaining a trained observation state prediction model and deploying it to an autonomous vehicle includes: obtaining the confidence corresponding to the observation data of multiple sensors; training an observation state prediction model using the observation data of multiple sensors, the corresponding confidence, and the observation state annotation information, obtaining a trained observation state prediction model and deploying it to an autonomous vehicle.

[0062] Considering that while each sensor outputs observation data, it also outputs the confidence information corresponding to the observation data. For example, an RTK positioning device can output the RTK positioning state. A differential solution state of the RTK positioning state indicates a higher accuracy of the RTK positioning data. A laser SLAM can output the confidence score of the laser SLAM positioning data. The higher the confidence score, the higher the accuracy of the laser SLAM positioning data.

[0063] Based on this, in the embodiments of the present application, when training the observation state prediction model, the confidence information output by multiple sensors can also be incorporated as prior knowledge into the model training process, thereby improving the convergence speed of the model.

[0064] In some embodiments of the present application, the training sample data for constructing the observation state prediction model includes: obtaining the observation data of multiple sensors on an autonomous vehicle in a target scenario and the true value data of a high-precision true value device, where the target scenario includes at least one of a tunnel, an urban canyon, and an open road; constructing the training sample data for the observation state prediction model according to the observation data of multiple sensors in the target scenario and the true value data of the high-precision true value device.

[0065] The selection and switching of multi-sensor observation information sources mainly occur in some specific road scenarios, such as tunnels, urban canyons, and open roads. In these scenarios, situations such as GNSS signal occlusion and feature degradation may occur, resulting in large errors in the observation data of the originally used main sensors. In other ordinary road scenarios, the RTK positioning device and laser SLAM positioning on an autonomous vehicle have relatively high-precision and stable positioning outputs.

[0066] Based on this, in order to improve the generalization ability of the model, in the embodiments of the present application, when collecting training sample data, different scenarios can be covered as much as possible. For example, data of scenarios such as tunnels, urban canyons, and open roads can be included. The observation state prediction model is trained with rich scenario data, thereby improving the generalization ability of the model. Of course, specifically which scenario data to collect can be flexibly set by those skilled in the art according to actual needs and will not be specifically limited here.

[0067] The embodiment of the present application also provides a method for predicting the observation state of multiple sensors, as follows Figure 2 As shown, it provides a schematic flowchart of a method for predicting the observation state of multiple sensors in the embodiment of the present application. The method for predicting the observation state of multiple sensors includes steps S210 to S220:

[0068] Step S210, obtaining the observation data of multiple sensors;

[0069] Step S220, inputting the observation data of multiple sensors into the observation state prediction model to obtain the observation state prediction result of multiple sensors;

[0070] Wherein, the observation state prediction model is trained based on the training method of the observation state prediction model of multiple sensors described in any one of the foregoing.

[0071] The prediction of the observation state of multiple sensors in the embodiment of the present application is realized based on the observation state prediction model of multiple sensors trained in the foregoing embodiments. When predicting the observation state of multiple sensors, the observation data collected by multiple sensors is obtained, and then the observation data collected by multiple sensors is input into the trained observation state prediction model of multiple sensors. The observation state prediction model of multiple sensors can directly output the observation state prediction result of multiple sensors, and specifically can output an observation state identifier. For example, if the output observation state identifier is Lidar-SLAM-2, it means that in the current observation state, laser SLAM positioning should be selected as the main observation information source.

[0072] Based on the pre-trained observation state prediction model of multiple sensors, the embodiment of the present application can accurately predict the observation state identifier of multiple sensors in real time, providing more accurate and reliable support for the selection of the observation information source of multiple sensors, especially for the selection of the observation information source of multiple sensors in special road scenarios. Compared with the existing method based on algorithm logic voting judgment, it has higher robustness and accuracy, and avoids the problem of inaccurate selection in critical situations.

[0073] In some embodiments of the present application, after inputting the observation data of multiple sensors into the observation state prediction model to obtain the observation state prediction result of multiple sensors, the method for predicting the observation state of multiple sensors further includes: dynamically adjusting the fusion weights corresponding to the observation information of each sensor according to the observation state prediction result of multiple sensors; performing fusion positioning according to the observation information of each sensor and the corresponding fusion weights to obtain a fusion positioning result.

[0074] As an application of the observation state prediction results of the multi-sensor obtained in the foregoing embodiments, the fusion positioning of the autonomous vehicle can be performed based on the predicted observation state identifiers of the multi-sensor. Specifically, the current main observation information source can be selected according to the predicted observation state identifiers of the multi-sensor, and then the fusion weight of the main observation information source can be appropriately increased, and the fusion weights of other observation information sources can be reduced. Then, based on the observation data of each adjusted observation information source and the corresponding fusion weights, the Kalman filter is input for filtering processing to obtain the current fusion positioning result of the autonomous vehicle.

[0075] The embodiment of the present application also provides a training device 300 for an observation state prediction model of a multi-sensor, as Figure 3 shown, which provides a schematic structural diagram of a training device for an observation state prediction model in an embodiment of the present application. The training device 300 for the observation state prediction model of the multi-sensor at least includes: a construction unit 310, a labeling unit 320, and a training unit 330, where:

[0076] The construction unit 310 is configured to construct training sample data for the observation state prediction model, and the training sample data includes observation data of the multi-sensor and true value data of the high-precision true value device;

[0077] The labeling unit 320 is configured to label the observation data of the multi-sensor according to the observation data of the multi-sensor and the true value data of the high-precision true value device to obtain observation state labeling information;

[0078] The training unit 330 is configured to train the observation state prediction model by using the observation data of the multi-sensor and the corresponding observation state labeling information, obtain the trained observation state prediction model, and deploy it to the autonomous vehicle.

[0079] In some embodiments of the present application, the labeling unit 320 is specifically configured to: calculate the error between the observation data of each sensor and the true value data respectively to obtain the observation error of each sensor; determine the observation state identifier corresponding to the observation data of the multi-sensor according to the observation error of each sensor.

[0080] In some embodiments of the present application, the training unit 330 is specifically configured to: obtain the confidence corresponding to the observation data of the multi-sensor; train the observation state prediction model by using the observation data of the multi-sensor, the corresponding confidence, and the observation state labeling information, obtain the trained observation state prediction model, and deploy it to the autonomous vehicle.

[0081] In some embodiments of the present application, the building unit 330 is specifically configured to: obtain the observation data of multiple sensors of an autonomous vehicle in a target scenario and the ground truth data of a high-precision ground truth device, where the target scenario includes at least one of a tunnel, an urban canyon, and an open road; construct the training sample data of the observation state prediction model according to the observation data of multiple sensors and the ground truth data of the high-precision ground truth device in the target scenario.

[0082] It can be understood that the above-mentioned training device for the observation state prediction model of multiple sensors can implement each step of the training method for the observation state prediction model of multiple sensors provided in the foregoing embodiments. The relevant explanations regarding the training method for the observation state prediction model of multiple sensors are applicable to the training device for the observation state prediction model of multiple sensors, and will not be elaborated here.

[0083] An embodiment of the present application further provides an observation state prediction device 400 for multiple sensors, as Figure 4 shown, which provides a schematic structural diagram of an observation state prediction device for multiple sensors in an embodiment of the present application. The observation state prediction device 400 for multiple sensors at least includes: an acquisition unit 410 and a prediction unit 420, where:

[0084] The acquisition unit 410 is configured to acquire the observation data of multiple sensors;

[0085] The prediction unit 420 is configured to input the observation data of multiple sensors into the observation state prediction model to obtain the observation state prediction result of multiple sensors;

[0086] Among them, the observation state prediction model is trained based on the foregoing training device for the observation state prediction model of multiple sensors.

[0087] In some embodiments of the present application, the observation state prediction device 400 for multiple sensors further includes: an adjustment unit, configured to dynamically adjust the fusion weights corresponding to the observation information of each sensor according to the observation state prediction result of multiple sensors after inputting the observation data of multiple sensors into the observation state prediction model to obtain the observation state prediction result of multiple sensors; a fusion positioning unit, configured to perform fusion positioning according to the observation information of each sensor and the corresponding fusion weights to obtain a fusion positioning result.

[0088] It can be understood that the above-mentioned observation state prediction device for multiple sensors can implement each step of the observation state prediction method for multiple sensors provided in the foregoing embodiments. The relevant explanations regarding the observation state prediction method for multiple sensors are applicable to the observation state prediction device for multiple sensors, and will not be elaborated here.

[0089] Figure 5It is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 5 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0090] The processor, network interface, and memory can be interconnected through an internal bus, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 5 only a bidirectional arrow is used in

[0091] but it does not mean that there is only one bus or one type of bus.

[0092] Memory, used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0092] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, and forms a training device for the observation state prediction model of multiple sensors or an observation state prediction device for multiple sensors at the logical level. The processor executes the program stored in the memory.

[0093] The above method executed by the training device for the observation state prediction model of multiple sensors disclosed in the embodiment as shown in the present application Figure 1 or Figure 2The method executed by the observation state prediction device of the multi-sensor disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed by the integrated logic circuit in the hardware of the processor or instructions in software form. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0094] The embodiments of the present application also propose a computer program product. The computer program product stores one or more programs. The one or more programs include instructions that, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 1 the method executed by the training device of the observation state prediction model of the multi-sensor in the illustrated embodiment, or execute Figure 2 the method executed by the observation state prediction device of the multi-sensor in the illustrated embodiment.

[0095] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The present invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each flow and / or block of the flowchart illustrations and / or block diagrams, and combinations of flows and / or blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processors of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart flow or flows and / or block or blocks. Figure 1 in a single flow or multiple flows and / or blocks Figure 1 or in a single block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the functions specified in the flowchart flow or flows and / or block or blocks. Figure 1 in a single flow or multiple flows and / or blocks Figure 1 or in a single block or multiple blocks.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and / or block or blocks. Figure 1 in a single flow or multiple flows and / or blocks Figure 1 or in a single block or multiple blocks.

[0099] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0100] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

[0101] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0102] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0103] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, system or computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] The above are only the embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A training method for a multi-sensor observation state prediction model, wherein: Multiple sensors are deployed on an autonomous driving vehicle, and a training method for an observation state prediction model of the multiple sensors includes: Constructing training sample data of the observation state prediction model, wherein the training sample data includes observation data of multiple sensors and true value data of high-precision true value equipment; The observation data of the multiple sensors are labeled according to the observation data of the multiple sensors and the true value data of the high-precision true value equipment to obtain the observation state labeling information; The observation data of multiple sensors and the corresponding observation state annotation information are used to train the observation state prediction model, and the trained observation state prediction model is obtained and deployed on the autonomous driving vehicle.

2. The method for training a multi-sensor observation state prediction model according to claim 1, wherein: The step of labeling the observation data of the multiple sensors according to the observation data of the multiple sensors and the true value data of the high-precision true value device to obtain the observation state labeling information includes: Calculating the error between the observation data of each sensor and the true value data respectively to obtain the observation error of each sensor; The observation state identifier corresponding to the observation data of the multiple sensors is determined according to the observation errors of the individual sensors.

3. The method for training a multi-sensor observation state prediction model according to claim 1, wherein: The method of training the observation state prediction model using the observation data of multiple sensors and the corresponding observation state annotation information, obtaining the trained observation state prediction model and deploying it on the autonomous driving vehicle includes: Obtain the confidence level corresponding to the observation data of multiple sensors; The observation data of multiple sensors, the corresponding confidence levels, and the observation state annotation information are used to train the observation state prediction model, and the trained observation state prediction model is obtained and deployed on the autonomous driving vehicle.

4. The method for training a multi-sensor observation state prediction model according to any one of claims 1 to 3, wherein: The training sample data for constructing the observation state prediction model includes: Acquire observation data of multiple sensors of the autonomous driving vehicle in a target scene and truth data of a high-precision truth device, wherein the target scene includes at least one of a tunnel, an urban canyon, and an open road; The training sample data of the observation state prediction model is constructed based on the observation data of multiple sensors in the target scene and the true value data of the high-precision true value equipment.

5. A method for predicting the observation state of multiple sensors, wherein: The multi-sensor observation state prediction method comprises: Obtain observation data from multiple sensors; Inputting the observation data of multiple sensors into the observation state prediction model to obtain the observation state prediction results of multiple sensors; Wherein, the observation state prediction model is trained based on the training method of the observation state prediction model of the multi-sensor according to any one of claims 1 to 4.

6. The method for predicting the observation state of multiple sensors as claimed in claim 5, wherein: After inputting the observation data of the multi-sensor into the observation state prediction model to obtain the observation state prediction result of the multi-sensor, the observation state prediction method of the multi-sensor further includes: Dynamically adjust the fusion weights corresponding to the observation information of each sensor according to the prediction results of the observation status of multiple sensors; Fusion positioning is performed based on the observation information of each sensor and the corresponding fusion weights to obtain the fusion positioning result.

7. A training device for a multi-sensor observation state prediction model, wherein: The training device of the multi-sensor observation state prediction model comprises: A construction unit, used to construct training sample data of an observation state prediction model, wherein the training sample data includes observation data of multiple sensors and true value data of a high-precision true value device; A labeling unit, used for labeling the observation data of the multiple sensors according to the observation data of the multiple sensors and the true value data of the high-precision true value device to obtain the observation state labeling information; The training unit is used to train the observation state prediction model using the observation data of multiple sensors and the corresponding observation state annotation information, obtain the trained observation state prediction model and deploy it on the autonomous driving vehicle.

8. A multi-sensor observation state prediction device, wherein: The multi-sensor observation state prediction device comprises: An acquisition unit, used for acquiring observation data of multiple sensors; A prediction unit, used for inputting the observation data of the multi-sensor into the observation state prediction model to obtain the observation state prediction result of the multi-sensor; Wherein, the observation state prediction model is trained based on the training device of the observation state prediction model of the multi-sensor according to claim 7.

9. An electronic device, comprising: processor; And a memory arranged to store computer executable instructions, which, when executed, cause the processor to execute the training method of the multi-sensor observation state prediction model of any one of claims 1 to 4, or execute the multi-sensor observation state prediction method of any one of claims 5 to 6.

10. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the training method of the multi-sensor observation state prediction model described in any one of claims 1 to 4, or executes the multi-sensor observation state prediction method described in any one of claims 5 to 6.

Images