Training method and device of universal perception model and automatic driving vehicle

By establishing neural network models for different types of sensors and configuring training thresholds and random numbers, and generating feature extraction and fusion models, the problem of poor universality of the perception model of autonomous driving vehicles is solved, and vehicles suitable for multiple sensor installation locations and types are realized , improves the applicability and efficiency of the model.

CN119004109APending Publication Date: 2024-11-22BEIJING XIAOMA ZHIXING TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411146409.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The perceptual model of autonomous driving vehicles is poor in versatility and cannot adapt to changes in different sensor installation positions and types, resulting in poor application of the model on different vehicles.

Method used

By establishing multiple neural network models, each model corresponds to different types of sensors, and configuring training thresholds and random numbers for it, selecting effective training data for machine learning, generating feature extraction models and feature fusion models, ensuring that the model is suitable for each Type of sensor installation location and type.

Benefits of technology

The universality of the perceptual model is improved, and it can be applied to vehicles of different sensor installation locations and types, improving the applicability and training efficiency of the model, and avoiding the trouble of retraining when sensor locations and types change.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004109A_ABST
    Figure CN119004109A_ABST
Patent Text Reader

Abstract

The invention provides a universal perception model training method and device and an automatic driving vehicle. The method comprises the following steps: establishing a plurality of first models configured with a first training threshold value, wherein a plurality of sensors in the same type are located at different positions of a vehicle; a plurality of training arrays are adopted to train a corresponding first model, so that the first model selects effective first training data according to a first training threshold value and the first random number to carry out machine learning, a plurality of feature extraction models are obtained, and the training arrays comprise the first training data of each sensor in the corresponding type; the first training data comprises sensor data and road condition feature information, the road condition feature information comprises a plurality of first grids and road condition features, the road condition features are located in the first grids mapped with the corresponding sensors, and the sensor data carries a first random number representing whether the sensor data is valid or not; and determining a universal perception model at least according to the plurality of feature extraction models.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and more specifically, to a training method, device, computer-readable storage medium, and autonomous driving vehicle for a general perception model. Background Art

[0002] A vehicle operating in an automated driving mode (e.g., without human control) can relieve occupants, particularly the driver, from some driving-related responsibilities. When operating in an automated driving mode, the vehicle can navigate various locations using onboard sensors, allowing the vehicle to be driven with minimal human interaction or in some cases without any passengers.

[0003] Motion planning and control are key operations in autonomous driving. The accuracy and efficiency of motion planning and control depend largely on the vehicle's sensors. However, different vehicle models have different sensor layouts, including the number of sensors, installation locations, and sensor types. Data collection and model training are required for vehicles with different sensor layouts, and the model generality is poor. Summary of the invention

[0004] The main purpose of this application is to provide a training method, device, computer-readable storage medium and autonomous driving vehicle for a universal perception model, so as to at least solve the problem of poor universality of the perception model of autonomous driving vehicles in the prior art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a training method for a universal perception model is provided, comprising: establishing multiple first models, wherein the first models are neural network models and are configured with a first training threshold, wherein the first training threshold is a threshold for determining whether input data is valid, wherein the first models correspond one-to-one to the type of sensor, and there are multiple sensors of the same type and they are located at different positions of the vehicle; using multiple training arrays to train the corresponding first models, so that the first models select valid first training data according to the first training threshold and a first random number for machine learning, and obtain multiple feature extraction models, wherein the training arrays correspond one-to-one to the types, and the training arrays include the first training data of each sensor of the corresponding type, the first training data include sensor data and road condition feature information, the road condition feature information includes multiple first grids and road condition features, the first grids are mapped to the positions of each sensor of the corresponding type, the road condition features are obtained by extracting features from the sensor data, the road condition features are located in the first grid mapped to the corresponding sensor, and the sensor data carries the first random number representing whether it is valid; and determining the universal perception model at least based on the multiple feature extraction models.

[0006] Optionally, a plurality of training arrays are used to train the corresponding first model, so that the first model selects valid first training data for machine learning according to the first training threshold and the first random number to obtain a plurality of feature extraction models, including: a first acquisition step, obtaining an initial training array corresponding to each of the types, the initial training array including a plurality of first initial training data, the first initial training data including the collected data corresponding to the sensor and the road condition feature information; a first configuration step, randomly configuring the first random number carried by each of the collected data to obtain the sensor data; a first training step, using a plurality of the first training data to train the corresponding first model, so that the first model selects the first random number corresponding to the first training threshold that is greater than the first training threshold according to the first training threshold. The first training data is the first target training data, and the first target training data is used for machine learning to obtain an initial feature extraction model; a first calculation step, calculating an evaluation index parameter of the initial feature extraction model to obtain a first parameter, and when the first parameter is less than a first preset value, looping the first configuration step, the first training step and the first calculation step until the first parameter is greater than or equal to the first preset value, wherein the first random number carried by at least part of the sensor data changes during two adjacent cycles; a first determination step, when the first parameter is greater than or equal to the first preset value, determining that the initial feature extraction model corresponding to the first parameter greater than or equal to the first preset value is the feature extraction model.

[0007] Optionally, calculating the evaluation index parameter of the initial feature extraction model to obtain the first parameter includes: acquiring the first training data that meets multiple first test requirements to obtain multiple first test data, and the multiple first test requirements include: all first random numbers corresponding to the type are greater than the first training threshold, and some of the first random numbers corresponding to the type are greater than the first training threshold; using multiple first test data, calculating the average accuracy of the initial feature extraction model to obtain the first parameter.

[0008] Optionally, before determining the universal perception model at least based on multiple feature extraction models, the method also includes: establishing a second model, the second model is a neural network model and is configured with a second training threshold, the second training threshold is a threshold for determining whether the input data is valid; using multiple second training data to train the second model, so that the second model selects valid second training data according to the second training threshold and the second random number for machine learning to obtain a feature fusion model, wherein the second training data includes feature fusion information and at least one pre-fusion feature information corresponding to the type, the pre-fusion feature information is the road condition feature information carrying the second random number representing whether it is valid, the feature fusion information includes multiple second grids and fusion features, the second grid has a mapping relationship with each of the first grids, the fusion feature is the information obtained after fusing at least one of the pre-fusion feature information, and the fusion feature is located in the second grid mapped to the first grid corresponding to the road condition feature information, and determining the universal perception model at least based on multiple feature extraction models includes: connecting the output ends of the multiple feature extraction models to the input end of the feature fusion model to obtain the universal perception model.

[0009] Optionally, a plurality of second training data are used to train the second model, so that the second model selects valid second training data for machine learning according to the second training threshold and the second random number to obtain a feature fusion model, including: a second acquisition step, obtaining second initial training data corresponding to each type of the sensor, the second initial training data including the feature fusion information and the road condition feature information corresponding to the plurality of types; a second configuration step, randomly configuring the second random number carried by each of the road condition feature information to obtain the feature information before fusion; a second training step, using a plurality of the second training data to train the second model, so that the second model selects the second training data corresponding to the second random number greater than the second training threshold according to the second training threshold. The data is the second target training data, and the second target training data is used for machine learning to obtain an initial feature fusion model; a second calculation step, calculating the evaluation index parameter of the initial feature fusion model to obtain a second parameter, and when the second parameter is less than a second preset value, looping the second configuration step, the second training step and the second calculation step until the second parameter is greater than or equal to the second preset value, wherein in two adjacent cycles, at least part of the second random number carried by the pre-fusion feature information changes; a second determination step, when the second parameter is greater than or equal to the second preset value, determining that the initial feature fusion model corresponding to the second parameter greater than or equal to the second preset value is the feature fusion model.

[0010] Optionally, calculating the evaluation index parameters of the initial feature fusion model to obtain the second parameters includes: acquiring the second training data that meets multiple second test requirements to obtain multiple second test data, the multiple second test requirements include: the second random numbers corresponding to each of the types are greater than the second training threshold, and the second random numbers corresponding to some types are greater than the second training threshold; using the second test data, calculating the average accuracy of the initial feature fusion model to obtain the second parameters.

[0011] Optionally, the types include at least some of the following: a visible light camera, an infrared camera, a laser radar, an ultrasonic radar, a millimeter wave radar, and an infrared radar, and the road condition characteristic information includes a BEV characteristic map.

[0012] According to another aspect of the present application, a training device for a universal perception model of a vehicle is provided, comprising: a first establishing unit, for establishing a plurality of first models, wherein the first model is a neural network model and is configured with a first training threshold, the first training threshold being a threshold for determining whether input data is valid, the first model corresponding one-to-one to the type of sensor, and the sensors of the same type are multiple and located at different positions of the vehicle; a first training unit, for training the corresponding first model using a plurality of training arrays, so that the first model selects valid first training data for machine learning according to the first training threshold and a first random number, and obtains a plurality of feature extraction models, wherein the training array corresponds one-to-one to the type, the training array includes the first training data of each sensor of the corresponding type, the first training data includes sensor data and road condition feature information, the road condition feature information includes a plurality of first grids and road condition features, the first grid is mapped to the position of each sensor of the corresponding type, the road condition feature is obtained by extracting features from the sensor data, the road condition feature is located in the first grid mapped to the corresponding sensor, and the sensor data carries the first random number representing whether it is valid; a determining unit, for determining the universal perception model at least based on a plurality of the feature extraction models.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0014] According to another aspect of the present application, an autonomous driving vehicle is provided, comprising: a vehicle body; at least one sensor mounted on the vehicle body; a controller of the vehicle body connected to at least one of the sensors, the controller comprising one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for executing any one of the methods described.

[0015] By applying the technical solution of the present application, the present application establishes corresponding first models for different types of sensors, configures a first training threshold for the first model, and configures a first random number for the training data of sensors located at different positions. In this way, when the first training data is used to train the first model of the corresponding type, the first model can determine whether the first training data corresponding to the sensors located at different positions of the vehicle are valid based on the size relationship between the first random number and the first training threshold. The first random number can be used to control whether the sensor data at different positions can participate in the model training, which is equivalent to training the first model with training data corresponding to different sensor installation positions, thereby realizing the training of perception models for vehicles with multiple sensor installation positions, ensuring that the obtained universal perception model can be applicable to the needs of vehicles with various sensor installation positions, and the model has good universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a training method for a general perception model provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of a flow chart of a method for training a general perception model provided according to an embodiment of the present application is shown;

[0019] Figure 3 A specific flow chart of a method for training a general perception model provided according to an embodiment of the present application is shown;

[0020] Figure 4 A structural block diagram of a general perception model training device provided according to an embodiment of the present application is shown.

[0021] The above drawings include the following reference numerals:

[0022] 102, processor; 104, memory; 106, transmission device; 108, input and output device; 110, first switch structure; 112, second switch structure. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the accompanying drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] As introduced in the background technology, the versatility of the perception model of the autonomous driving vehicle in the prior art is relatively poor. To solve the above technical problems, the embodiments of the present application provide a training method, device, computer-readable storage medium and autonomous driving vehicle for a universal perception model.

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a general perception model training method according to an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the training method of the general perception model in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the method described is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The transmission device 106 is used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so that it can communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a training method for a general perception model running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] Figure 2 FIG. 1 is a flow chart of a method for training a universal perception model according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0032] Step S201, establishing a plurality of first models, wherein the first models are neural network models and are configured with a first training threshold, the first training threshold being a threshold for determining whether input data is valid, the first models corresponding to sensor types one by one, and there are multiple sensors of the same type located at different positions of the vehicle;

[0033] Specifically, the types include at least some of the following: visible light camera, infrared camera, laser radar, ultrasonic radar, millimeter wave radar, infrared radar. One sensor type corresponds to one first model. Multiple sensors of each type are installed at different positions of the vehicle, such as the front, rear, and side of the vehicle, so as to achieve forward, rear, surround, side, and interior vision effects.

[0034] Step S202, using a plurality of training arrays to train the corresponding first model, so that the first model selects valid first training data for machine learning according to the first training threshold and the first random number, and obtains a plurality of feature extraction models, wherein the training array corresponds to the type one by one, the training array includes the first training data of each sensor under the corresponding type, the first training data includes sensor data and road condition feature information, the road condition feature information includes a plurality of first grids and road condition features, the first grids are mapped to the positions of each sensor under the corresponding type, the road condition features are obtained by extracting features from the sensor data, the road condition features are located in the first grid mapped to the corresponding sensor, and the sensor data carries the first random number indicating whether it is valid;

[0035] Specifically, the first training data are all historical data of the vehicle. Each sensor corresponds to at least one first training data, and the sensor data is the data collected by the sensor. The mapping relationship between the first grid and the position of the sensor can be a one-to-one correspondence between the first grid and the position of the sensor, or one first grid corresponds to the positions of multiple sensors, or multiple first grids correspond to the position of one sensor. Since there is a mapping relationship between the first grid and the position of the sensor, the first grid and the sensor data, as well as the road condition features extracted from the sensor data, also have the same mapping relationship. The first model determines which first training data are valid and which first training data are invalid by comparing the size relationship between the first training threshold and the first random number. The first model does not use invalid first training data for machine learning. The first model after training is a feature extraction model. The input of the feature extraction model after training is sensor data, and the output is road condition feature information.

[0036] Step S203, determining the universal perception model at least based on the multiple feature extraction models.

[0037] Through the embodiment, firstly, a plurality of first models corresponding to the types of sensors are established, each of which is a neural network model and is configured with a first training threshold. There are multiple sensors of each type and they are located at different positions of the vehicle. Then, a training array including a plurality of first training data is used to train the first model. The first training data includes sensor data carrying a first random number and road condition feature information. The first model selects valid first training data from the training array for machine learning based on the first random number and the first training threshold. The learned first model obtains a feature extraction model. Finally, a general perception model is obtained based at least on these feature extraction models. Compared with the technical problem in the prior art that the perception models of vehicles with different sensor installation positions cannot be universal, the present application establishes corresponding first models for different types of sensors, configures a first training threshold for the first model, and configures a first random number for the training data of sensors located at different positions. In this way, when the first training data is used to train the first model of the corresponding type, the first model can determine whether the first training data corresponding to the sensors located at different positions of the vehicle is valid based on the size relationship between the first random number and the first training threshold. The first random number can be used to control whether the sensor data at different positions can participate in the model training, which is equivalent to training the first model with training data corresponding to different sensor installation positions, thereby realizing the training of perception models for vehicles with multiple sensor installation positions, ensuring that the obtained universal perception model can be applicable to the needs of vehicles with various sensor installation positions, and the model has good universality.

[0038] It should be noted that this application configures the positions of various sensors of the same type by collecting the sensor installation positions of the current vehicle, so that these configured positions can basically include the sensor positions of all current vehicles, thereby ensuring that the obtained general perception model has high applicability. For example, for the type of camera, some vehicles have cameras installed on both sides of the front and rear of the vehicle, some vehicles have cameras installed on the roof, and some vehicles have cameras installed on both sides of the front and both sides of the vehicle. Then, in this application, the positions where sensors such as cameras are set include: both sides of the front, both sides of the rear, both sides of the vehicle and the roof, and the first training data corresponding to the collected camera includes sensor data of each position.

[0039] In the solution of the present application, by setting the first training threshold and the first random number, it is equivalent to setting a switch structure for the sensor at each position. When the first training data is valid, it means that the sensor corresponding to the first training data is turned on, and when the first training data is invalid, it means that the sensor corresponding to the first training data is turned off. Through the first random number, the switch control of the sensors at each position is realized, so that the training data of the vehicle scenes with different sensor installation positions can be obtained, so that the training model can adapt to different vehicle scenes.

[0040] Specifically, determining the universal perception model at least based on the plurality of feature extraction models includes: obtaining the universal perception model including the plurality of feature extraction models based on the plurality of feature extraction models. In this embodiment, the universal perception model includes feature extraction models of multiple types of sensors, which are used to extract features from sensor data to obtain road condition information where the vehicle is located.

[0041] In an optional solution, step S202: using multiple training arrays to train the corresponding first model, so that the first model selects valid first training data according to the first training threshold and the first random number for machine learning, and obtains multiple feature extraction models, including:

[0042] Step S2021: a first acquisition step, acquiring an initial training array corresponding to each of the types, wherein the initial training array includes a plurality of first initial training data, and the first initial training data includes the collected data corresponding to the sensor and the road condition feature information;

[0043] Specifically, the road condition characteristic information is the road condition information of the vehicle corresponding to the collected data.

[0044] Step S2022: a first configuration step, randomly configuring the first random number carried by each of the collected data to obtain the sensor data;

[0045] Specifically, the collected data after configuring the first random number is the sensor data, and the sensor data and the road condition characteristic information constitute the first training data. By controlling the size of the first random number, it is determined whether the sensor data is valid, which is equivalent to configuring a switch structure for the sensor at each position. When the first random number corresponding to the first training data is greater than the first training threshold, it indicates that the sensor corresponding to the first training data is turned on, and the sensor data at this position participates in the training of the model, and when the first random number corresponding to the first training data is less than or equal to the first training threshold, it indicates that the sensor corresponding to the first training data is turned off, and the sensor data at this position does not participate in the training of the model.

[0046] Step S2023: a first training step, using a plurality of the first training data to train the corresponding first model, so that the first model selects the first training data corresponding to the first random number greater than the first training threshold as the first target training data according to the first training threshold, and uses the first target training data to perform machine learning to obtain an initial feature extraction model;

[0047] Specifically, the first target training data is valid first training data in a training array corresponding to the first model.

[0048] Step S2024: a first calculation step, calculating an evaluation index parameter of the initial feature extraction model to obtain a first parameter, and when the first parameter is less than a first preset value, loopingly executing the first configuration step, the first training step, and the first calculation step until the first parameter is greater than or equal to the first preset value, wherein the first random number carried by at least part of the sensor data changes during two adjacent loop processes;

[0049] Specifically, the evaluation index parameter can be any suitable index for evaluating model performance, such as accuracy, precision, recall rate, F1 value, ROC curve and AUC value.

[0050] Step S2025: A first determination step, in a case where the first parameter is greater than or equal to the first preset value, determining that the initial feature extraction model corresponding to the first parameter that is greater than or equal to the first preset value is the feature extraction model.

[0051] In the embodiment, when the initial feature extraction model does not meet the evaluation index parameters of the model, the first configuration step, the first training step and the first calculation step are executed through training to randomly adjust the first random number size of each first training data, thereby obtaining a training array for vehicle models with different sensor positions, further achieving the effect of training the model using a combination of sensor data at different positions, further ensuring that the feature extraction model trained by the training array can be adapted to vehicle models with various sensor installation positions, and further ensuring that the universal perception model obtained according to the feature extraction model has a high degree of universality.

[0052] Further, calculating the evaluation index parameter of the initial feature extraction model to obtain the first parameter includes: obtaining the first training data that meets multiple first test requirements to obtain multiple first test data, wherein the multiple first test requirements include: all first random numbers corresponding to the type are greater than the first training threshold, that is, the first training data corresponding to the sensors at each position are valid, which is equivalent to the sensors at each position being turned on; some of the first random numbers corresponding to the type are greater than the first training threshold, that is, the first training data corresponding to the sensors at some positions are valid, and the first training data corresponding to the sensors at some positions are invalid, which is equivalent to turning on the sensors at some positions; using multiple first test data, calculating the average accuracy of the initial feature extraction model to obtain the first parameter. In this embodiment, by calculating the average accuracy of the model corresponding to the case where all sensors are turned on and the case where some sensors are turned on, the performance of the initial feature extraction model is evaluated, which further ensures that the obtained feature extraction model can meet the vehicle model requirements of different sensor installation positions, further avoids the problem of needing to retrain the perception model of the vehicle when the sensor installation position changes, further ensures the high versatility of the perception model, and further ensures the high training efficiency of the perception model.

[0053] In this embodiment, before step S202: determining the universal perception model at least according to the plurality of feature extraction models, the method further includes:

[0054] Step S204: establishing a second model, wherein the second model is a neural network model and is configured with a second training threshold, wherein the second training threshold is a threshold for determining whether input data is valid;

[0055] Step S205: Use multiple second training data to train the second model, so that the second model selects valid second training data according to the second training threshold and the second random number for machine learning, and obtains a feature fusion model, wherein the second training data includes feature fusion information and at least one pre-fusion feature information corresponding to the type, the pre-fusion feature information is the road condition feature information carrying the second random number representing whether it is valid, the feature fusion information includes multiple second grids and fusion features, the second grids are mapped to each of the first grids, the fusion feature is information obtained after fusing at least one of the pre-fusion feature information, and the fusion feature is located in the second grid mapped to the first grid corresponding to the road condition feature information.

[0056] Specifically, when there is only one feature before fusion, it means that the vehicle has only one type of sensor, and the corresponding fusion feature is the feature before fusion; when there are multiple features before fusion, it means that the vehicle has multiple types of sensors, and the corresponding fusion feature is the fusion of multiple features before fusion. The second random number is equivalent to setting a switch structure for the sensor type. When the second random number indicates that the second training data is valid, it means that the switch structure of the sensor of this type is turned on, and when the second random number indicates that the second training data is invalid, it means that the switch structure of the sensor of this type is turned off. Sensor types with closed switch structures do not participate in feature fusion.

[0057] The embodiment takes into account the different types of sensors in different vehicle layouts, establishes a second model for feature fusion of feature information collected by different sensors, configures a second training threshold for the second model, and configures a second random number for training data of different types of sensors. In this way, when the second training data is used to train the second model, the second model can determine whether different types of sensor data can participate in model training based on the size relationship between the second random number and the second training threshold, which is equivalent to training the second model using training models corresponding to different sensor types, thereby realizing the training of perception models for vehicles with multiple sensor installation types. The universal perception model obtained in this way can be applicable not only to vehicles with different sensor installation positions, but also to vehicles with different sensor types, further ensuring that the universal perception model of the present application can be applicable to vehicles with different sensor layouts.

[0058] Based on the embodiment, the general perception model is determined at least according to the plurality of feature extraction models, including: connecting the output ends of the plurality of feature extraction models with the input ends of the feature fusion model to obtain the general perception model. That is, firstly extracting features of the corresponding type of sensor data through the feature extraction model, and then fusing features of at least one type of sensor features through the feature fusion model, thereby obtaining feature fusion information to characterize the actual road condition of the vehicle.

[0059] It should be noted that the input of the trained feature fusion model is various types of pre-fusion feature information, and the output is feature fusion information.

[0060] Specifically, step S205: using a plurality of second training data to train the second model, so that the second model selects valid second training data according to the second training threshold and the second random number for machine learning, to obtain a feature fusion model, including:

[0061] Step S2051: a second acquisition step, acquiring second initial training data corresponding to each type of the sensor, wherein the second initial training data includes the feature fusion information and the road condition feature information corresponding to a plurality of the types;

[0062] Specifically, the feature fusion information is information obtained by fusion of feature information collected by various types of sensors, and is also information used to characterize vehicle road conditions.

[0063] Step S2052: a second configuration step, randomly configuring the second random number carried by each of the road condition feature information to obtain the feature information before fusion;

[0064] Specifically, the road condition feature information after configuring the second random number is the feature information before fusion, and the feature information before fusion and the feature fusion information constitute the second training data. By controlling the size of the second random number, it is determined whether the feature information before fusion is valid, which is equivalent to configuring a switch structure for each type of sensor. When the second random number corresponding to the second training data is greater than the second training threshold, it indicates that the sensor type corresponding to the second training data is turned on, and the sensor data of this type participates in the training of the model, and when the second random number corresponding to the second training data is less than or equal to the second training threshold, it indicates that the sensor type corresponding to the second training data is turned off, and the sensor data of this type does not participate in the training of the model.

[0065] Step S2053: a second training step, using a plurality of the second training data to train the second model, so that the second model selects the second training data corresponding to the second random number greater than the second training threshold as the second target training data according to the second training threshold, and uses the second target training data to perform machine learning to obtain an initial feature fusion model;

[0066] Specifically, the second target training data is valid second training data.

[0067] Step S2054: a second calculation step, calculating the evaluation index parameter of the initial feature fusion model to obtain a second parameter, and when the second parameter is less than a second preset value, looping the second configuration step, the second training step and the second calculation step until the second parameter is greater than or equal to the second preset value, wherein the second random number carried by at least part of the feature information before fusion changes during two adjacent cycles;

[0068] Specifically, the evaluation index parameter can be any suitable index for evaluating model performance, such as accuracy, precision, recall rate, F1 value, ROC curve and AUC value.

[0069] Step S2055: a second determination step, in the case where the second parameter is greater than or equal to the second preset value, determining the initial feature fusion model corresponding to the second parameter greater than or equal to the second preset value as the feature fusion model.

[0070] In the embodiment, when the initial feature fusion model does not meet the evaluation index parameters of the model, the second configuration step, the second training step and the second calculation step are executed through training to randomly adjust the second random number size of each second training data, thereby obtaining training data of vehicle models equipped with different sensor types, further achieving the effect of training the model using different types of sensor data combinations, further ensuring that the feature fusion model trained by the training data can be adapted to vehicle models equipped with various sensor type combinations (such as equipped with cameras and lidars, only equipped with cameras, only equipped with lidars, and equipped with cameras, lidars and millimeter-wave radars), further ensuring that the universal perception model obtained according to the feature fusion model has high universality.

[0071] Further, calculating the evaluation index parameter of the initial feature fusion model to obtain the second parameter includes: obtaining the second training data that meets multiple second test requirements to obtain multiple second test data, wherein the multiple second test requirements include: the second random numbers corresponding to each type are greater than the second training threshold, that is, the second training data corresponding to each type of sensor are valid, which is equivalent to all types of sensors being turned on, and the second random numbers corresponding to some types are greater than the second training threshold, that is, the second training data corresponding to some types of sensors are valid, and the second training data corresponding to some types of sensors are invalid, which is equivalent to only some types of sensors being turned on; using the second test data, calculating the average accuracy of the initial feature fusion model to obtain the second parameter. In this embodiment, by calculating the average accuracy of the model corresponding to the case where each type of sensor is turned on and the case where some types of sensors are turned on, the performance of the initial feature fusion model is evaluated, which further ensures that the obtained feature fusion model can meet the vehicle model requirements of different sensor type combinations, further avoids the problem of needing to retrain the perception model of the vehicle when the sensor type changes, further ensures the high versatility of the perception model, and further ensures the high training efficiency of the perception model.

[0072] In an embodiment of the present application, the road condition feature information includes a BEV feature map. Specifically, during the training process of the first model, the sensor data is firstly extracted, and then the extracted feature information is converted into a feature space, thereby obtaining a BEV feature map.

[0073] The first model and the second model of the present application may be any suitable neural network model, such as a convolutional neural network model, etc. Those skilled in the art may flexibly set the specific model types of the first model and the second model according to actual needs.

[0074] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the training method of the universal perception model of the present application will be described in detail below in combination with specific embodiments.

[0075] This embodiment relates to a specific method for training a universal perception model, such as Figure 3 As shown, the following steps are included:

[0076] Step S1: Establish a first model corresponding to a sensor type, and obtain a plurality of first training data of the model, wherein the first model is configured with a first training threshold, and the first switch structure 110 of the sensor at each position of the type is controlled by randomly configuring a first random number carried by the first training data, that is, when the first random number is greater than the first training threshold, it is equivalent to turning on the sensor at the corresponding position; on the contrary, when the first random number is less than or equal to the first training threshold, it is equivalent to turning off the sensor at the corresponding position;

[0077] Step S2: training the first model using a plurality of first training data until the trained first model meets the evaluation index parameters of the model, thereby obtaining a feature extraction model;

[0078] Step S3: looping steps S1 and S2 to obtain a feature extraction model for each sensor type;

[0079] Step S4: Establish a second model, and obtain a plurality of second training data of the model, wherein the second model is configured with a second training threshold, and the second switch structure 112 of the sensor type is controlled by randomly configuring a second random number carried by the second training data, that is, when the second random number is greater than the second training threshold, it is equivalent to turning on the corresponding type of sensor; on the contrary, when the second random number is less than or equal to the second training threshold, it is equivalent to turning off the corresponding type of sensor;

[0080] Step S5: training the second model using a plurality of second training data until the trained second model meets the evaluation index parameters of the model, thereby obtaining a feature fusion model;

[0081] Step S6: Combining the feature extraction model with the feature fusion model so that the output of the feature extraction model is connected to the input of the feature fusion model to obtain the general perception model.

[0082] It should be noted that the Figure 3The training method of the general perception model is only exemplarily shown in which there are three types of sensors (i.e., a first type of sensor, a second type of sensor, and a third type of sensor) and there are three installation positions of each type of sensor (each type of sensor 1, sensor 2, and sensor 3 corresponds to one position respectively). The types of sensors and the number of installation positions of each type of sensor are not limited to Figure 3 The 3 values ​​shown in FIG. 1 and 10 should be set by a technician in this field according to the actual vehicle layout.

[0083] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0084] The embodiment of the present application also provides a training device for a universal perceptual model. It should be noted that the training device for a universal perceptual model in the embodiment of the present application can be used to execute the training method for a universal perceptual model provided in the embodiment of the present application. The device is used to implement the embodiments and preferred implementations, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0085] The following introduces the training device of the general perception model provided in the embodiment of the present application.

[0086] Figure 4 Schematic diagram of a training device for a universal perception model according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0087] A first establishing unit 10 is used to establish a plurality of first models, wherein the first model is a neural network model and is configured with a first training threshold, the first training threshold being a threshold for determining whether input data is valid, the first model corresponds to a type of sensor in a one-to-one manner, and there are multiple sensors of the same type and they are located at different positions of the vehicle;

[0088] Specifically, the types include at least some of the following: visible light camera, infrared camera, laser radar, ultrasonic radar, millimeter wave radar, infrared radar. One sensor type corresponds to one first model. Multiple sensors of each type are installed at different positions of the vehicle, such as the front, rear, and side of the vehicle, so as to achieve forward, rear, surround, side, and interior vision effects.

[0089] A first training unit 20 is used to train the corresponding first model using a plurality of training arrays, so that the first model selects valid first training data for machine learning according to the first training threshold and the first random number to obtain a plurality of feature extraction models, wherein the training array corresponds to the type one by one, the training array includes the first training data of each sensor under the corresponding type, the first training data includes sensor data and road condition feature information, the road condition feature information includes a plurality of first grids and road condition features, the first grids are mapped to the positions of each sensor under the corresponding type, the road condition features are obtained by extracting features from the sensor data, the road condition features are located in the first grid mapped to the corresponding sensor, and the sensor data carries the first random number indicating whether it is valid;

[0090] Specifically, the first training data are all historical data of the vehicle. Each sensor corresponds to at least one first training data, and the sensor data is the data collected by the sensor. The mapping relationship between the first grid and the position of the sensor can be a one-to-one correspondence between the first grid and the position of the sensor, or one first grid corresponds to the positions of multiple sensors, or multiple first grids correspond to the position of one sensor. Since there is a mapping relationship between the first grid and the position of the sensor, the first grid and the sensor data, as well as the road condition features extracted from the sensor data, also have the same mapping relationship. The first model determines which first training data are valid and which first training data are invalid by comparing the size relationship between the first training threshold and the first random number. The first model does not use invalid first training data for machine learning. The first model after training is a feature extraction model. The input of the feature extraction model after training is sensor data, and the output is road condition feature information.

[0091] The determination unit 30 is used to determine the universal perception model at least based on the multiple feature extraction models.

[0092] In the embodiment, a plurality of first models corresponding to the types of sensors are established by a first establishing unit, each of the first models is a neural network model and is configured with a first training threshold, and there are multiple sensors of each type and they are located at different positions of the vehicle; the first model is trained by a first training unit using a training array including a plurality of first training data, the first training data including sensor data carrying a first random number and road condition feature information, the first model selects valid first training data from the training array for machine learning based on the first random number and the first training threshold, and the learned first model obtains a feature extraction model; the general perception model is obtained by a determining unit based at least on these feature extraction models. Compared with the technical problem in the prior art that the perception models of vehicles with different sensor installation positions cannot be universal, the present application establishes corresponding first models for different types of sensors, configures a first training threshold for the first model, and configures a first random number for the training data of sensors located at different positions. In this way, when the first training data is used to train the first model of the corresponding type, the first model can determine whether the first training data corresponding to the sensors located at different positions of the vehicle is valid based on the size relationship between the first random number and the first training threshold. The first random number can be used to control whether the sensor data at different positions can participate in the model training, which is equivalent to training the first model with training data corresponding to different sensor installation positions, thereby realizing the training of perception models for vehicles with multiple sensor installation positions, ensuring that the obtained universal perception model can be applicable to the needs of vehicles with various sensor installation positions, and the model has good universality.

[0093] It should be noted that this application configures the positions of various sensors of the same type by collecting the sensor installation positions of the current vehicle, so that these configured positions can basically include the sensor positions of all current vehicles, thereby ensuring that the obtained general perception model has high applicability. For example, for the type of camera, some vehicles have cameras installed on both sides of the front and rear of the vehicle, some vehicles have cameras installed on the roof, and some vehicles have cameras installed on both sides of the front and both sides of the vehicle. Then, in this application, the positions where sensors such as cameras are set include: both sides of the front, both sides of the rear, both sides of the vehicle and the roof, and the first training data corresponding to the collected camera includes sensor data of each position.

[0094] In the solution of the present application, by setting the first training threshold and the first random number, it is equivalent to setting a switch structure for the sensor at each position. When the first training data is valid, it means that the sensor corresponding to the first training data is turned on, and when the first training data is invalid, it means that the sensor corresponding to the first training data is turned off. Through the first random number, the switch control of the sensors at each position is realized, so that the training data of the vehicle scenes with different sensor installation positions can be obtained, so that the training model can adapt to different vehicle scenes.

[0095] Specifically, the determination unit includes: an obtaining module, which is used to obtain the general perception model including multiple feature extraction models according to multiple feature extraction models. In this embodiment, the general perception model includes feature extraction models of multiple types of sensors, which are used to extract features from sensor data to obtain road condition information where the vehicle is located.

[0096] In an optional solution, the first training unit includes:

[0097] A first acquisition module, used in a first acquisition step, to acquire an initial training array corresponding to each of the types, wherein the initial training array includes a plurality of first initial training data, and the first initial training data includes the collected data corresponding to the sensor and the road condition feature information;

[0098] Specifically, the road condition characteristic information is the road condition information of the vehicle corresponding to the collected data.

[0099] A first configuration module, used in a first configuration step, to randomly configure the first random number carried by each of the collected data to obtain the sensor data;

[0100] Specifically, the collected data after configuring the first random number is the sensor data, and the sensor data and the road condition characteristic information constitute the first training data. By controlling the size of the first random number, it is determined whether the sensor data is valid, which is equivalent to configuring a switch structure for the sensor at each position. When the first random number corresponding to the first training data is greater than the first training threshold, it indicates that the sensor corresponding to the first training data is turned on, and the sensor data at this position participates in the training of the model, and when the first random number corresponding to the first training data is less than or equal to the first training threshold, it indicates that the sensor corresponding to the first training data is turned off, and the sensor data at this position does not participate in the training of the model.

[0101] A first training module, used in a first training step, uses a plurality of the first training data to train the corresponding first model, so that the first model selects the first training data corresponding to the first random number greater than the first training threshold as the first target training data according to the first training threshold, and uses the first target training data to perform machine learning to obtain an initial feature extraction model;

[0102] Specifically, the first target training data is valid first training data in a training array corresponding to the first model.

[0103] A first calculation module, used for a first calculation step, calculating an evaluation index parameter of the initial feature extraction model to obtain a first parameter, and when the first parameter is less than a first preset value, cyclically executing the first configuration step, the first training step, and the first calculation step until the first parameter is greater than or equal to the first preset value, wherein the first random number carried by at least part of the sensor data changes during two adjacent cycles;

[0104] Specifically, the evaluation index parameter can be any suitable index for evaluating model performance, such as accuracy, precision, recall rate, F1 value, ROC curve and AUC value.

[0105] The first determination module is used for the first determination step, and when the first parameter is greater than or equal to the first preset value, determines that the initial feature extraction model corresponding to the first parameter greater than or equal to the first preset value is the feature extraction model.

[0106] In the embodiment, when the initial feature extraction model does not meet the evaluation index parameters of the model, the first configuration step, the first training step and the first calculation step are executed through training to randomly adjust the first random number size of each first training data, thereby obtaining a training array for vehicle models with different sensor positions, further achieving the effect of training the model using a combination of sensor data at different positions, further ensuring that the feature extraction model trained by the training array can be adapted to vehicle models with various sensor installation positions, and further ensuring that the universal perception model obtained according to the feature extraction model has a high degree of universality.

[0107] Further, the first calculation module includes: a first acquisition submodule, used to acquire the first training data that meets multiple first test requirements, and obtain multiple first test data, wherein the multiple first test requirements include: all first random numbers corresponding to the type are greater than the first training threshold, that is, the first training data corresponding to the sensors at each position are valid, which is equivalent to the sensors at each position being turned on; some of the first random numbers corresponding to the type are greater than the first training threshold, that is, the first training data corresponding to the sensors at some positions are valid, and the first training data corresponding to the sensors at some positions are invalid, which is equivalent to turning on the sensors at some positions; the first calculation submodule, used to use the multiple first test data to calculate the average accuracy of the initial feature extraction model to obtain the first parameter. In this embodiment, the performance of the initial feature extraction model is evaluated by calculating the average accuracy of the model when all sensors are turned on and when some sensors are turned on, which further ensures that the obtained feature extraction model can meet the vehicle model requirements of different sensor installation positions, further avoids the problem of retraining the perception model of the vehicle when the sensor installation position changes, further ensures the high versatility of the perception model, and further ensures the high training efficiency of the perception model.

[0108] In this embodiment, the device further includes:

[0109] A second establishing unit is used to establish a second model before determining the universal perception model based on at least a plurality of the feature extraction models, wherein the second model is a neural network model and is configured with a second training threshold, and the second training threshold is a threshold for determining whether the input data is valid;

[0110] The second training unit is used to train the second model with multiple second training data, so that the second model selects valid second training data according to the second training threshold and the second random number for machine learning to obtain a feature fusion model, wherein the second training data includes feature fusion information and at least one pre-fusion feature information corresponding to the type, the pre-fusion feature information is the road condition feature information carrying the second random number representing whether it is valid, the feature fusion information includes multiple second grids and fusion features, the second grids are mapped to each of the first grids, the fusion feature is the information obtained after fusing at least one of the pre-fusion feature information, and the fusion feature is located in the second grid mapped to the first grid corresponding to the road condition feature information.

[0111] Specifically, when there is only one feature before fusion, it means that the vehicle has only one type of sensor, and the corresponding fusion feature is the feature before fusion; when there are multiple features before fusion, it means that the vehicle has multiple types of sensors, and the corresponding fusion feature is the fusion of multiple features before fusion. The second random number is equivalent to setting a switch structure for the sensor type. When the second random number indicates that the second training data is valid, it means that the switch structure of the sensor of this type is turned on, and when the second random number indicates that the second training data is invalid, it means that the switch structure of the sensor of this type is turned off. Sensor types with closed switch structures do not participate in feature fusion.

[0112] The embodiment takes into account the different types of sensors in different vehicle layouts, establishes a second model for feature fusion of feature information collected by different sensors, configures a second training threshold for the second model, and configures a second random number for training data of different types of sensors. In this way, when the second training data is used to train the second model, the second model can determine whether different types of sensor data can participate in model training based on the size relationship between the second random number and the second training threshold, which is equivalent to training the second model using training models corresponding to different sensor types, thereby realizing the training of perception models for vehicles with multiple sensor installation types. The universal perception model obtained in this way can be applicable not only to vehicles with different sensor installation positions, but also to vehicles with different sensor types, further ensuring that the universal perception model of the present application can be applicable to vehicles with different sensor layouts.

[0113] Based on the embodiment, the determination unit includes: a connection module, which is used to connect the output ends of the plurality of feature extraction models with the input end of the feature fusion model to obtain the general perception model. That is, the feature extraction model is used to extract features of the corresponding type of sensor data, and then the feature fusion model is used to fuse features of at least one type of sensor features, thereby obtaining feature fusion information to characterize the actual road condition of the vehicle.

[0114] It should be noted that the input of the trained feature fusion model is various types of pre-fusion feature information, and the output is feature fusion information.

[0115] Specifically, the second training unit includes:

[0116] A second acquisition module, used in a second acquisition step, to acquire second initial training data corresponding to each type of the sensor, wherein the second initial training data includes the feature fusion information and the road condition feature information corresponding to a plurality of the types;

[0117] Specifically, the feature fusion information is information obtained by fusion of feature information collected by various types of sensors, and is also information used to characterize vehicle road conditions.

[0118] A second configuration module, used in a second configuration step, randomly configures the second random number carried by each of the road condition feature information to obtain the feature information before fusion;

[0119] Specifically, the road condition feature information after configuring the second random number is the feature information before fusion, and the feature information before fusion and the feature fusion information constitute the second training data. By controlling the size of the second random number, it is determined whether the feature information before fusion is valid, which is equivalent to configuring a switch structure for each type of sensor. When the second random number corresponding to the second training data is greater than the second training threshold, it indicates that the sensor type corresponding to the second training data is turned on, and the sensor data of this type participates in the training of the model, and when the second random number corresponding to the second training data is less than or equal to the second training threshold, it indicates that the sensor type corresponding to the second training data is turned off, and the sensor data of this type does not participate in the training of the model.

[0120] a second training module, used for a second training step, using a plurality of the second training data to train the second model, so that the second model selects the second training data corresponding to the second random number greater than the second training threshold as the second target training data according to the second training threshold, and uses the second target training data to perform machine learning to obtain an initial feature fusion model;

[0121] Specifically, the second target training data is valid second training data.

[0122] A second calculation module is used for a second calculation step, calculating the evaluation index parameter of the initial feature fusion model to obtain a second parameter, and when the second parameter is less than a second preset value, cyclically executing the second configuration step, the second training step, and the second calculation step until the second parameter is greater than or equal to the second preset value, wherein the second random number carried by at least part of the pre-fusion feature information changes during two adjacent cycles;

[0123] Specifically, the evaluation index parameter can be any suitable index for evaluating model performance, such as accuracy, precision, recall rate, F1 value, ROC curve and AUC value.

[0124] The second determination module is used for the second determination step, and when the second parameter is greater than or equal to the second preset value, determines that the initial feature fusion model corresponding to the second parameter greater than or equal to the second preset value is the feature fusion model.

[0125] In the embodiment, when the initial feature fusion model does not meet the evaluation index parameters of the model, the second configuration step, the second training step and the second calculation step are executed through training to randomly adjust the second random number size of each second training data, thereby obtaining training data of vehicle models equipped with different sensor types, further achieving the effect of training the model using different types of sensor data combinations, further ensuring that the feature fusion model trained by the training data can be adapted to vehicle models equipped with various sensor type combinations (such as equipped with cameras and lidars, only equipped with cameras, only equipped with lidars, and equipped with cameras, lidars and millimeter-wave radars), further ensuring that the universal perception model obtained according to the feature fusion model has high universality.

[0126] Further, the second calculation module includes: a second acquisition submodule, which is used to acquire the second training data that meets multiple second test requirements, and obtain multiple second test data, wherein the multiple second test requirements include: the second random numbers corresponding to each type are greater than the second training threshold, that is, the second training data corresponding to each type of sensor are valid, which is equivalent to all types of sensors being turned on, and the second random numbers corresponding to some types are greater than the second training threshold, that is, the second training data corresponding to some types of sensors are valid, and the second training data corresponding to some types of sensors are invalid, which is equivalent to only some types of sensors being turned on; the second calculation submodule is used to use the second test data to calculate the average accuracy of the initial feature fusion model to obtain the second parameter. In this embodiment, the performance of the initial feature fusion model is evaluated by calculating the average accuracy of the model when each type of sensor is turned on and when some types of sensors are turned on, which further ensures that the obtained feature fusion model can meet the vehicle model requirements of different sensor type combinations, further avoids the problem of retraining the perception model of the vehicle when the sensor type changes, further ensures the high versatility of the perception model, and further ensures the high training efficiency of the perception model.

[0127] In an embodiment of the present application, the road condition feature information includes a BEV feature map. Specifically, during the training process of the first model, the sensor data is firstly extracted, and then the extracted feature information is converted into a feature space, thereby obtaining a BEV feature map.

[0128] The first model and the second model of the present application may be any suitable neural network model, such as a convolutional neural network model, etc. Those skilled in the art may flexibly set the specific model types of the first model and the second model according to actual needs.

[0129] The training device of the general perception model includes a processor and a memory, wherein the first establishment unit, the first training unit, and the determination unit are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0130] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be provided, and the problem of poor universality of the perception model of the autonomous driving vehicle in the prior art can be at least solved by adjusting the kernel parameters.

[0131] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0132] An embodiment of the present application provides an autonomous driving vehicle, comprising: a vehicle body; at least one sensor mounted on the vehicle body; a controller of the vehicle body connected to at least one of the sensors, the controller comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for executing any one of the methods described.

[0133] An embodiment of the present application provides a computer-readable storage medium, which includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the training method of the general perception model.

[0134] An embodiment of the present application provides a processor, which is used to run a program, wherein the training method of the universal perception model is executed when the program is running.

[0135] An embodiment of the present application provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0136] Step S201, establishing a plurality of first models, wherein the first models are neural network models and are configured with a first training threshold, the first training threshold being a threshold for determining whether input data is valid, the first models corresponding to sensor types one by one, and there are multiple sensors of the same type located at different positions of the vehicle;

[0137] Step S202, using a plurality of training arrays to train the corresponding first model, so that the first model selects valid first training data for machine learning according to the first training threshold and the first random number, and obtains a plurality of feature extraction models, wherein the training array corresponds to the type one by one, the training array includes the first training data of each sensor under the corresponding type, the first training data includes sensor data and road condition feature information, the road condition feature information includes a plurality of first grids and road condition features, the first grids are mapped to the positions of each sensor under the corresponding type, the road condition features are obtained by extracting features from the sensor data, the road condition features are located in the first grid mapped to the corresponding sensor, and the sensor data carries the first random number indicating whether it is valid;

[0138] Step S203, determining the universal perception model at least based on the multiple feature extraction models.

[0139] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0140] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement at least the following method steps:

[0141] Step S201, establishing a plurality of first models, wherein the first models are neural network models and are configured with a first training threshold, the first training threshold being a threshold for determining whether input data is valid, the first models corresponding to sensor types one by one, and there are multiple sensors of the same type located at different positions of the vehicle;

[0142] Step S202, using a plurality of training arrays to train the corresponding first model, so that the first model selects valid first training data for machine learning according to the first training threshold and the first random number, and obtains a plurality of feature extraction models, wherein the training array corresponds to the type one by one, the training array includes the first training data of each sensor under the corresponding type, the first training data includes sensor data and road condition feature information, the road condition feature information includes a plurality of first grids and road condition features, the first grids are mapped to the positions of each sensor under the corresponding type, the road condition features are obtained by extracting features from the sensor data, the road condition features are located in the first grid mapped to the corresponding sensor, and the sensor data carries the first random number indicating whether it is valid;

[0143] Step S203, determining the universal perception model at least based on the multiple feature extraction models.

[0144] Obviously, those skilled in the art should understand that the modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0145] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0146] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0149] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0150] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0151] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0152] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0153] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a general perception model, characterized in that: include: Establishing a plurality of first models, wherein the first models are neural network models and are configured with a first training threshold, the first training threshold being a threshold for determining whether input data is valid, the first models corresponding to types of sensors one by one, and there are multiple sensors of the same type and located at different positions of the vehicle; Using a plurality of training arrays to train the corresponding first model, so that the first model selects valid first training data for machine learning according to the first training threshold and the first random number, and obtains a plurality of feature extraction models, wherein the training array corresponds to the type one by one, the training array includes the first training data of each sensor under the corresponding type, the first training data includes sensor data and road condition feature information, the road condition feature information includes a plurality of first grids and road condition features, the first grids are mapped to the positions of each sensor under the corresponding type, the road condition features are obtained by extracting features from the sensor data, the road condition features are located in the first grid mapped to the corresponding sensor, and the sensor data carries the first random number indicating whether it is valid; The general perception model is determined based on at least a plurality of the feature extraction models.

2. The method according to claim 1, characterized in that The first model corresponding to the first training data set is trained using a plurality of training data sets, so that the first model selects valid first training data for machine learning according to the first training threshold and the first random number, and obtains a plurality of feature extraction models, including: A first acquisition step, acquiring an initial training array corresponding to each of the types, wherein the initial training array includes a plurality of first initial training data, and the first initial training data includes the collected data corresponding to the sensor and the road condition feature information; A first configuration step is to randomly configure the first random number carried by each of the collected data to obtain the sensor data; a first training step, using a plurality of the first training data to train the corresponding first model, so that the first model selects the first training data corresponding to the first random number greater than the first training threshold as the first target training data according to the first training threshold, and uses the first target training data to perform machine learning to obtain an initial feature extraction model; A first calculation step, calculating an evaluation index parameter of the initial feature extraction model to obtain a first parameter, and when the first parameter is less than a first preset value, loopingly executing the first configuration step, the first training step, and the first calculation step until the first parameter is greater than or equal to the first preset value, wherein the first random number carried by at least part of the sensor data changes during two adjacent loop processes; The first determination step is to determine, when the first parameter is greater than or equal to the first preset value, the initial feature extraction model corresponding to the first parameter greater than or equal to the first preset value as the feature extraction model.

3. The method according to claim 2, characterized in that Calculating the evaluation index parameter of the initial feature extraction model to obtain the first parameter includes: Acquire the first training data that meets multiple first test requirements to obtain multiple first test data, wherein the multiple first test requirements include: all first random numbers corresponding to the type are greater than the first training threshold, and some of the first random numbers corresponding to the type are greater than the first training threshold; The first parameters are obtained by using a plurality of the first test data to calculate an average accuracy of the initial feature extraction model.

4. The method according to claim 1, characterized in that: Before determining the universal perception model at least based on the plurality of feature extraction models, the method further comprises: Establishing a second model, wherein the second model is a neural network model and is configured with a second training threshold, wherein the second training threshold is a threshold for determining whether input data is valid; The second model is trained using a plurality of second training data, so that the second model selects valid second training data according to the second training threshold and the second random number for machine learning, and obtains a feature fusion model, wherein the second training data includes feature fusion information and at least one pre-fusion feature information corresponding to the type, the pre-fusion feature information is the road condition feature information carrying the second random number representing whether the information is valid, the feature fusion information includes a plurality of second grids and fusion features, the second grids are mapped to each of the first grids, the fusion feature is information obtained after fusing at least one of the pre-fusion feature information, and the fusion feature is located in the second grid mapped to the first grid corresponding to the road condition feature information, Determining the general perception model at least according to the plurality of feature extraction models comprises: The output ends of the plurality of feature extraction models are connected to the input end of the feature fusion model to obtain the universal perception model.

5. The method according to claim 4, characterized in that The second model is trained using a plurality of second training data, so that the second model selects valid second training data for machine learning according to the second training threshold and the second random number to obtain a feature fusion model, including: A second acquisition step, acquiring second initial training data corresponding to each type of the sensor, wherein the second initial training data includes the feature fusion information and the road condition feature information corresponding to a plurality of the types; A second configuration step is to randomly configure the second random number carried by each of the road condition feature information to obtain the feature information before fusion; a second training step, using a plurality of the second training data to train the second model, so that the second model selects the second training data corresponding to the second random number greater than the second training threshold as the second target training data according to the second training threshold, and uses the second target training data to perform machine learning to obtain an initial feature fusion model; A second calculation step, calculating an evaluation index parameter of the initial feature fusion model to obtain a second parameter, and when the second parameter is less than a second preset value, loopingly executing the second configuration step, the second training step, and the second calculation step until the second parameter is greater than or equal to the second preset value, wherein the second random number carried by at least part of the pre-fusion feature information changes during two adjacent cycles; The second determination step is to determine, when the second parameter is greater than or equal to the second preset value, the initial feature fusion model corresponding to the second parameter greater than or equal to the second preset value as the feature fusion model.

6. The method according to claim 5, characterized in that Calculating the evaluation index parameter of the initial feature fusion model to obtain the second parameter includes: Acquire the second training data that meets multiple second test requirements to obtain multiple second test data, wherein the multiple second test requirements include: the second random numbers corresponding to each of the types are greater than the second training threshold, and the second random numbers corresponding to some types are greater than the second training threshold; The second test data is used to calculate the average accuracy of the initial feature fusion model to obtain the second parameter.

7. The method according to any one of claims 1 to 6, characterized in that The types include at least some of the following: a visible light camera, an infrared camera, a laser radar, an ultrasonic radar, a millimeter wave radar, and an infrared radar, and the road condition characteristic information includes a BEV characteristic map.

8. A training device for a universal perception model of a vehicle, characterized in that: include: A first establishing unit is used to establish a plurality of first models, wherein the first models are neural network models and are configured with a first training threshold, the first training threshold being a threshold for determining whether input data is valid, the first models correspond to sensor types one by one, and there are multiple sensors of the same type and they are located at different positions of the vehicle; A first training unit is used to train the corresponding first model using a plurality of training arrays, so that the first model selects valid first training data for machine learning according to the first training threshold and a first random number to obtain a plurality of feature extraction models, wherein the training array corresponds to the type one by one, the training array includes the first training data of each sensor under the corresponding type, the first training data includes sensor data and road condition feature information, the road condition feature information includes a plurality of first grids and road condition features, the first grids are mapped to the positions of each sensor under the corresponding type, the road condition features are obtained by extracting features from the sensor data, the road condition features are located in the first grid mapped to the corresponding sensor, and the sensor data carries the first random number indicating whether it is valid; A determination unit is used to determine the general perception model based on at least a plurality of the feature extraction models.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. An autonomous driving vehicle, characterized in that: include: Vehicle body; at least one sensor mounted on the vehicle body; The controller of the vehicle body is connected to at least one of the sensors, and the controller includes one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of claims 1 to 7.

Citation Information

Patent Citations

  • Multi-perception fusion sensor bionic method and system

    CN118395152A

  • Generating environmental parameters based on sensor data using machine learning

    US20200209858A1

  • System and method for addressing overfitting in a neural network

    US9406017B2

  • Method for inferring lane, and method and apparatus for training lane inference model

    WO2022001432A1