Perception model training method and device, vehicle and storage medium

Abstract

The invention provides a perceptual model training method and device, a vehicle and a storage medium, and the method comprises the steps: obtaining first sample perceptual data collected by the vehicle, the first sample perceptual data having a corresponding first perceptual feature, and determining second sample perceptual data related to key path information, the second sample perceptual data having a corresponding second perceptual feature, the second sample sensing data has a corresponding second sensing feature, determining that the first sample sensing data is target sample sensing data according to the first sensing feature and the second sensing feature, and determining that the weight value of the target sample sensing data in the target scene is greater than a weight threshold value. And training the initial perception model according to the second sample perception data and the target sample perception data to obtain the target perception model, thereby giving full play to the data value of the collected sample perception data, and effectively improving the training effect of the vehicle end perception model based on the sample perception data.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to a training method, device, vehicle and storage medium for a perception model. Background Art

[0002] In the field of autonomous driving technology, the vehicle-side perception model provides the autonomous driving system with powerful environmental perception and understanding capabilities, bringing higher safety, stability and efficiency. It enables autonomous driving vehicles to accurately perceive and respond to the surrounding traffic environment, thereby achieving intelligent autonomous driving.

[0003] Therefore, how to improve the training effect of the vehicle-side perception model to obtain a vehicle-side perception model with better performance has become a key research direction. Summary of the invention

[0004] To overcome the problems existing in the related art, the present disclosure provides a method, an apparatus, a vehicle, a non-transitory computer-readable storage medium storing computer instructions, and a computer program product for training a perception model.

[0005] The training method of the perception model proposed in the embodiment of the first aspect of the present disclosure includes: acquiring first sample perception data collected by a vehicle, wherein the first sample perception data has a corresponding first perception feature; determining second sample perception data related to critical path information, wherein the second sample perception data has a corresponding second perception feature; determining, based on the first perception feature and the second perception feature, that the first sample perception data is target sample perception data, wherein a weight value of the target sample perception data in the target scene is greater than a weight threshold; and training an initial perception model based on the second sample perception data and the target sample perception data to obtain a target perception model.

[0006] The perception method based on the perception model proposed in the second aspect embodiment of the present disclosure includes: acquiring the data to be perceived collected by the vehicle; inputting the data to be perceived into the target perception model trained by the perception model training method proposed in the first aspect embodiment above, so as to obtain the target perception result output by the target perception model.

[0007] The training device of the perception model proposed in the third aspect embodiment of the present disclosure includes: a first acquisition module, used to acquire first sample perception data collected by a vehicle, wherein the first sample perception data has a corresponding first perception feature; a first determination module, used to determine second sample perception data related to critical path information, wherein the second sample perception data has a corresponding second perception feature; a second determination module, used to determine that the first sample perception data is target sample perception data based on the first perception feature and the second perception feature, wherein the weight value of the target sample perception data in the target scene is greater than a weight threshold; a training module, used to train an initial perception model based on the second sample perception data and the target sample perception data to obtain a target perception model.

[0008] The perception device based on the perception model proposed in the fourth embodiment of the present disclosure includes: a second acquisition module for acquiring the data to be perceived collected by the vehicle; a processing module for inputting the data to be perceived into the target perception model trained by the training device of the perception model proposed in the third embodiment, so as to obtain the target perception result output by the target perception model

[0009] The vehicle proposed in the fifth aspect embodiment of the present disclosure includes: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: implement the training method of the perception model proposed in the first aspect embodiment of the present disclosure.

[0010] The sixth aspect embodiment of the present disclosure proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the training method of the perception model proposed in the first aspect embodiment of the present disclosure, or implements the perception method based on the perception model proposed in the second aspect embodiment of the present disclosure.

[0011] The seventh aspect embodiment of the present disclosure proposes a computer program product. When the instructions in the computer program product are executed by a processor, the training method of the perception model proposed in the first aspect embodiment of the present disclosure is executed, or the perception method based on the perception model proposed in the second aspect embodiment of the present disclosure is implemented.

[0012] The training method of the perception model provided by the embodiment of the present disclosure may include the following beneficial effects: by acquiring first sample perception data collected by the vehicle, wherein the first sample perception data has a corresponding first perception feature, and determining second sample perception data related to the critical path information, wherein the second sample perception data has a corresponding second perception feature, and then determining, based on the first perception feature and the second perception feature, that the first sample perception data is target sample perception data, wherein a weight value of the target sample perception data in the target scene is greater than a weight threshold, and then training an initial perception model based on the second sample perception data and the target sample perception data to obtain a target perception model, thereby being able to give full play to the data value of the collected sample perception data, and being able to effectively improve the training effect of the vehicle-side perception model based on the sample perception data.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a flowchart of a method for training a perception model according to some embodiments of the present disclosure;

[0016] Figure 2 is a flowchart of a method for training a perception model according to some embodiments of the present disclosure;

[0017] Figure 3 is a flow chart of a perception method based on a perception model according to some embodiments of the present disclosure;

[0018] Figure 4 is a block diagram of a training device for a perception model according to some embodiments of the present disclosure;

[0019] Figure 5 is a block diagram of a perception device based on a perception model according to some embodiments of the present disclosure;

[0020] Figure 6 is a block diagram of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0021] Some embodiments of the present disclosure will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications and equivalents of the methods, devices and / or systems described herein will become apparent after understanding the present disclosure. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as becomes apparent after understanding the present disclosure, except for operations that must be performed in a specific order. In addition, for clarity and brevity, descriptions of features known in the art may be omitted.

[0022] The embodiments described in some embodiments of the present disclosure below do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0023] Figure 1 is a flowchart of a method for training a perception model according to some embodiments of the present disclosure, such as Figure 1 As shown, the following steps are included:

[0024] S101: Acquire first sample perception data collected by a vehicle, wherein the first sample perception data has a corresponding first perception feature.

[0025] Among them, the data collected by the vehicle for training the vehicle-side perception model can be called the first sample perception data. The first sample perception data can be specifically, for example, image data collected by the vehicle-side camera, lidar data collected by the vehicle-side lidar, millimeter-wave radar data collected by the vehicle-side millimeter-wave radar, IMU data collected by the vehicle-side inertial measurement unit (IMU), etc., without limitation.

[0026] That is to say, in the embodiments of the present disclosure, to obtain the first sample perception data collected by the vehicle, the corresponding data acquisition devices, such as cameras, lidars and IMUs, may be pre-set for the vehicle, and then, during the vehicle driving process, the camera may be controlled to collect image data in the vehicle driving scene, and / or the lidar may be controlled to collect lidar data in the vehicle driving scene, and / or the IMU may be controlled to collect IMU data in the vehicle driving scene, and the aforementioned collected image data in the vehicle driving scene, and / or lidar data, and / or IMU data may be collectively used as the first sample perception data, and there is no limitation to this.

[0027] Among them, the first sample perception data may have a corresponding data feature, which may be called a first perception feature.

[0028] In the disclosed embodiment, after obtaining the first sample perception data collected by the vehicle, the first sample perception data can be cleaned (for example, missing value filling, outlier removal, etc.), and a feature extraction algorithm (for example, principal component analysis, oriented gradient histogram algorithm, etc., without limitation) can be used to process the first sample perception data to obtain a first perception feature corresponding to the first sample perception data.

[0029] S102: Determine second sample perception data related to the critical path information, wherein the second sample perception data has a corresponding second perception feature.

[0030] Among them, the sample perception data related to the critical path information in the sample perception data used to train the vehicle-side perception model can be called the second sample perception data.

[0031] In some embodiments, obtaining the second sample perception data related to the critical path information may be by screening the initial sample perception data, that is, the initial sample perception data may be analyzed for sample perception data containing data features related to the critical path information. For example, the sample perception data may be analyzed to see whether it contains entities related to the critical path information (e.g., roads, paths), and the sample perception data containing data features related to the critical path information may be used as the second sample perception data, without limitation.

[0032] Among them, the second sample perception data may have a corresponding data feature, which may be called a second perception feature.

[0033] In the disclosed embodiment, after obtaining the second sample perception data collected by the vehicle, the second sample perception data can be cleaned (for example, missing value filling, outlier removal, etc.), and a feature extraction algorithm (for example, principal component analysis, oriented gradient histogram algorithm, etc., without limitation) can be used to process the second sample perception data to obtain a second perception feature corresponding to the second sample perception data.

[0034] In the disclosed embodiment, the initial perception model includes: a path planning sub-model, and the path planning sub-model can be used to determine the best driving path for the vehicle based on the data collected by the vehicle.

[0035] Optionally, in some embodiments, sample perception data used when the path planning sub-model predicts key path information is determined, and the used sample perception data is used as second sample perception data.

[0036] Among them, the key path information predicted by the path planning sub-model refers to the driving path predicted by the path planning sub-model for the vehicle when there is user intervention, and the driving path when the user does not drive according to the path predicted by the path planning sub-model.

[0037] That is to say, in the embodiments of the present disclosure, when it is determined that the output of the path planning sub-model is the key path information, the input data of the path planning sub-model can be determined, and the input data can be used as the second sample perception data, thereby improving the comprehensiveness and data value of the second sample perception data, and continuing to train the path planning sub-model based on the sample perception data.

[0038] S103: Determine, based on the first perceptual feature and the second perceptual feature, that the first sample perceptual data is the target sample perceptual data, wherein a weight value of the target sample perceptual data in the target scene is greater than a weight threshold.

[0039] After determining the first perceptual feature and the second perceptual feature, the embodiment of the present disclosure can determine, based on the first perceptual feature and the second perceptual feature, that the first sample perceptual data is the target sample perceptual data, wherein a weight value of the target sample perceptual data in the target scene is greater than a weight threshold.

[0040] Among them, the target scenario refers to the application scenario of the vehicle-side perception model. The application scenario can be specifically, for example, a target detection scenario, a target recognition scenario, etc., and there is no restriction on this.

[0041] Among them, the weight value of the target sample perception data in the target scene refers to the role played by the target perception data in executing the task in the target scene when the task in the target scene is executed based on the target sample perception data (that is, it can be, for example, the accuracy of target detection when performing a target detection task in a target detection scenario based on certain target perception data; it can also be, for example, the recall rate of the vehicle-side model when training the vehicle-side model in the target scene based on the target perception data; it can also be, for example, the accuracy of path planning when using the target perception data for the path planning task in the path planning scenario, etc., without limitation).

[0042] In the disclosed embodiment, since the second perceptual feature is a data feature of the second sample perceptual data related to the critical path information, the first perceptual feature and the second perceptual feature can be combined to determine whether the first sample perceptual data is the target perceptual data in the target scene whose weight value is greater than the weight threshold.

[0043] In some embodiments, determining that the first sample perception data is the target sample perception data based on the first perceptual feature and the second perceptual feature may be performed by determining the feature similarity between the first perceptual feature and the second perceptual feature, and determining that the first sample perception data is the target sample perception data when the feature similarity is greater than a feature similarity threshold.

[0044] In other embodiments, the first perceptual feature can also be used in a task in the target scene, and then, based on the task execution result of the first perceptual feature in the target scene, it is determined whether the first sample perception data is the target sample perception data. For example, the first perceptual feature can be used in a target detection task in a target detection scene, and the corresponding target detection result can be determined. When the target detection result is the same as the reference target detection result, the first sample perception data corresponding to the first perceptual feature is determined to be the target sample perception data. There is no limitation on this.

[0045] S104: Train an initial perception model according to the second sample perception data and the target sample perception data to obtain a target perception model.

[0046] Among them, the untrained vehicle-side perception model obtained at the initial stage when the perception model training method starts to execute can be called the initial perception model. The initial perception model can be, for example, a model for performing target detection tasks, a model for performing semantic segmentation tasks, a model for performing three-dimensional perception tasks, and other models for one or more vehicle assisted driving tasks, without any restriction.

[0047] After determining that the first sample perception data is the target sample perception data, the embodiment of the present disclosure can train an initial perception model based on the second sample perception data and the target sample perception data until the initial perception model converges, and use the converged initial perception model as the target perception model.

[0048] In some embodiments, the initial perception model is trained based on the second sample perception data and the target sample perception data to obtain the target perception model. The second sample perception data and the target sample perception data can be used together as sample perception data for training the initial perception model, and the labeled perception results corresponding to the sample perception data can be determined. Then, the second sample perception data and the target sample perception data can be input into the initial perception model to obtain the predicted perception results output by the initial perception model. Then, the loss value between the predicted perception results and the labeled perception results can be determined in combination with a preset loss function, and the initial perception model can be supervised based on the loss value for iterative training. When the loss value is less than a loss threshold, it is determined that the initial perception model has converged, and the converged initial perception model is used as the target perception model.

[0049] In other embodiments, the initial perception model is trained based on the second sample perception data and the target sample perception data to obtain the target perception model. Alternatively, the second sample perception data and the target sample perception data are input into the initial perception model to obtain the predicted perception results output by the initial perception model, and the predicted perception results are used in downstream vehicle-side tasks of model training (for example, target detection tasks, path planning tasks, etc., without limitation). When the downstream vehicle-side task determines that the predicted perception results meet the requirements in the downstream vehicle-side task, the initial perception model is determined to converge, and the converged initial perception model is used as the target perception model, without limitation.

[0050] In the disclosed embodiment, by acquiring first sample perception data collected by the vehicle, wherein the first sample perception data has a corresponding first perception feature, and determining second sample perception data related to the critical path information, wherein the second sample perception data has a corresponding second perception feature, and then determining based on the first perception feature and the second perception feature that the first sample perception data is target sample perception data, wherein a weight value of the target sample perception data in the target scene is greater than a weight threshold, and then training an initial perception model based on the second sample perception data and the target sample perception data to obtain a target perception model, thereby being able to give full play to the data value of the collected sample perception data and being able to effectively improve the training effect of the vehicle-side perception model based on the sample perception data.

[0051] Figure 2 It is a flowchart of a method for training a perception model according to some embodiments of the present disclosure.

[0052] like Figure 2 As shown, the training method of the perception model includes the following steps:

[0053] S201: Acquire first sample perception data collected by the vehicle, wherein the first sample perception data has a corresponding first perception feature.

[0054] S202: Determine second sample perception data related to the critical path information, wherein the second sample perception data has a corresponding second perception feature.

[0055] The specific description of S201 - S202 can be found in the above embodiment, which will not be repeated here.

[0056] S203: Determine the feature similarity between the first perceptual feature and the second perceptual feature.

[0057] The feature similarity between the first perceptual feature and the second perceptual feature may be, for example, the Euclidean distance between the first perceptual feature and the second perceptual feature, the cosine similarity between the first feature and the second feature, etc., and there is no limitation to this.

[0058] That is to say, in the embodiments of the present disclosure, determining the feature similarity between the first perceptual feature and the second perceptual feature may be determining the Euclidean distance between the first perceptual feature and the second perceptual feature, or determining the cosine similarity between the first perceptual feature and the second perceptual feature, and using the aforementioned determined Euclidean distance or cosine similarity as the feature similarity, without limitation.

[0059] S204: Acquire multiple task processing score values ​​related to the first perception feature, wherein the task processing score values ​​are used for the path planning sub-model to plan the path.

[0060] Among them, the task processing score value is used for the path planning sub-model to plan the path. The task processing score value can be used to quantitatively describe the path planning effect of the path planning sub-model. That is, the higher the task processing score value, the better the path planning effect of the path planning sub-model. Conversely, the lower the task processing score value, the worse the path planning effect of the path planning sub-model.

[0061] Optionally, in some embodiments, obtaining multiple task processing score values ​​related to the first perceptual feature may be inputting the first perceptual feature into each initial task processing sub-model to obtain multiple task processing score values ​​output by the multiple initial task processing sub-models respectively.

[0062] Among them, the initial task processing sub-model can be used to perform corresponding tasks during vehicle driving, such as target detection tasks, target tracking tasks, etc., without any restriction.

[0063] Among them, the task processing score value can be used to describe the task completion status of the initial task processing sub-model when processing the first perception feature to achieve the execution of the corresponding task.

[0064] That is to say, in the embodiment of the present disclosure, the first perceptual feature can be input into each initial task processing sub-model to obtain multiple task processing score values ​​output by multiple initial task processing sub-models respectively, so that the task processing score values ​​can be accurately obtained.

[0065] S205: Determine, based on feature similarity and / or multiple task processing score values, that the first sample perception data is the target sample perception data.

[0066] After determining the feature similarity and / or multiple task processing score values, the embodiment of the present disclosure can determine that the first sample perception data is the target sample perception data based on the feature similarity and / or multiple task processing score values.

[0067] In some embodiments, the first sample perception data is determined to be the target sample perception data based on feature similarity and / or multiple task processing score values. The first sample perception data is determined to be the target sample perception data when the feature similarity is greater than a similarity threshold, and / or the number of multiple task processing score values ​​that are greater than a score threshold is greater than a quantity threshold.

[0068] In other embodiments, the first sample perception data is determined to be the target sample perception data based on feature similarity and / or multiple task processing score values, or by weighted summing the feature similarity values ​​and multiple task processing score values ​​based on a preset weight coefficient, and when the value obtained by the weighted sum is greater than a preset value, the first sample perception data is determined to be the target sample perception data. There is no limitation on this.

[0069] Optionally, in some embodiments, determining whether the first sample perception data is the target sample perception data based on feature similarity and / or multiple task processing score values ​​may be performed by determining whether the feature similarity is less than a similarity threshold to obtain a first result, and / or determining a score weight value corresponding to each task processing score value, performing weighted averaging processing on multiple task processing score values ​​based on the score weight value, and determining whether the value obtained by weighted processing is less than the task score threshold to obtain a second result, and determining that the first sample perception data is the target sample perception data based on the first result and / or the second result.

[0070] In the disclosed embodiment, the feature similarity may be compared with a similarity threshold to obtain a corresponding comparison result, which is the first result.

[0071] Each task processing score value may have a preset weight coefficient, which is the score weight value.

[0072] That is to say, in the embodiment of the present disclosure, a weighted average of multiple task processing score values ​​can be performed according to the score weight value to obtain a weighted processing value, and the weighted processing value can be compared with the task score threshold to obtain a corresponding comparison result, which is the second result.

[0073] After determining the first result and the second result, the embodiment of the present disclosure can determine that the first sample perception data is the target sample perception data based on the first result and / or the second result, that is, a first reference result corresponding to the first result can be obtained in advance, and a second reference result corresponding to the second result can be obtained, and when the first result is the same as the first reference result, and / or the second result is the same as the second reference result, it is determined that the first sample perception data is the target sample perception data.

[0074] Optionally, in some embodiments, determining that the first sample perception data is the target sample perception data is based on the first result and / or the second result. When the feature similarity of the first result is less than a similarity threshold, and / or the value obtained by weighted processing of the second result is less than a task score threshold, determining that the first sample perception data is the target sample perception data.

[0075] That is to say, in the embodiments of the present disclosure, when the feature similarity of the first result is less than the similarity threshold, and / or the value obtained by weighted processing of the second result is less than the task score threshold, it is determined that the first sample perception data is the target sample perception data. Thus, it is possible to accurately determine whether the first sample perception data is the target sample perception data, thereby effectively ensuring the accuracy of the target sample perception data.

[0076] S206: Input the target sample perception data into the initial feature extraction sub-model to obtain the target perception features output by the initial feature extraction sub-model and corresponding to the target sample perception data.

[0077] In the disclosed embodiment, the initial perception model includes: an initial feature extraction sub-model.

[0078] Among them, the initial feature extraction sub-model can be used to extract data features of sample perception data.

[0079] Among them, the target sample perception data may have a corresponding data feature, which is the target perception feature.

[0080] That is to say, in the embodiments of the present disclosure, the target sample perception data can be input into the initial feature extraction sub-model, and the initial feature extraction sub-model can perform feature extraction on the target sample perception data to obtain the target perception features corresponding to the target sample perception data output by the initial feature extraction sub-model.

[0081] S207: Perform at least one iterative training on each of the multiple initial task processing sub-models according to the second perception feature and the target perception feature of the second sample perception data, until the number of iterative training is less than the number threshold, and each of the multiple task processing sub-models obtained through training meets the first convergence condition, and use each of the multiple task processing sub-models obtained through training as a plurality of target task processing sub-models.

[0082] Among them, the preset number of iterative training times pre-set for the initial task processing sub-model can be called the number threshold. The number threshold can be adaptively set in combination with the model training requirements of the initial task sub-model, and there is no restriction on this.

[0083] Among them, the first convergence condition can be a model convergence condition pre-set for the initial task sub-model. The first convergence condition can be, for example, that the downstream task meets the standard and the index of the trained model in the evaluation set is higher than the index threshold, and there is no restriction on this.

[0084] Among them, when the number of iterative training of the initial task sub-model is less than the number threshold, and multiple trained task processing sub-models all meet the first convergence condition, the trained task processing sub-model is the target task processing sub-model.

[0085] That is to say, in the embodiments of the present disclosure, multiple initial task processing sub-models may be iteratively trained at least once according to the second perception features and target perception features of the second sample perception data, and when the number of iterative training is less than a threshold number and multiple task processing sub-models obtained through training all meet the first convergence condition, the multiple task processing sub-models obtained through training may be respectively used as multiple target task processing sub-models.

[0086] S208: Generate a target perception model based on the initial feature extraction sub-model and multiple target task processing sub-models.

[0087] In the embodiment of the present disclosure, after the initial task processing sub-model is trained until the model converges to obtain the target task processing sub-model, a target perception model can be generated based on the initial feature extraction sub-model and multiple target task processing sub-models.

[0088] That is to say, in the embodiment of the present disclosure, the initial feature extraction sub-model and multiple target task processing sub-models can be connected based on the connection layer to generate a target perception model.

[0089] S209: If the number of iterative training reaches the threshold number, and at least one task processing sub-model obtained through training does not meet the first convergence condition, the initial feature extraction sub-model is iteratively trained at least once according to the second sample perception data and the target sample perception data until the feature extraction sub-model obtained through training meets the second convergence condition, and the feature extraction sub-model obtained through training is used as the target feature extraction sub-model.

[0090] Among them, the second convergence condition can be a model convergence condition pre-set for the initial feature extraction sub-model. The second convergence condition can be, for example, that the downstream task meets the standard, the number of training times reaches a threshold, etc., and there is no restriction on this.

[0091] In the disclosed embodiment, when the trained feature extraction sub-model satisfies the second convergence condition, the trained feature extraction sub-model can be referred to as a target feature extraction sub-model, and there is no limitation on this.

[0092] In the embodiment of the present disclosure, multiple initial task processing sub-models may be iteratively trained at least once according to the second perception features and target perception features of the second sample perception data, and when the number of training generations reaches a threshold number and at least one trained task processing sub-model does not meet the first convergence condition, the initial feature extraction sub-model is iteratively trained at least once according to the second sample perception data and the target sample perception data until the trained feature extraction sub-model meets the second convergence condition, and the trained feature extraction sub-model is used as the target feature extraction sub-model.

[0093] S210: updating the second perceptual feature based on the target feature extraction sub-model to obtain a second updated perceptual feature, and updating the target perceptual feature based on the target feature extraction sub-model to obtain a target updated perceptual feature.

[0094] In the disclosed embodiment, after the target feature extraction sub-model is obtained through training, the second perceptual feature can be updated based on the target feature extraction sub-model to obtain a second updated perceptual feature, and the target perceptual feature can be updated based on the target feature extraction sub-model to obtain a target updated perceptual feature, thereby enabling timely updating of training parameters and effectively ensuring the training effect of the perceptual model.

[0095] S211: Iteratively train multiple initial task processing sub-models according to the second updated perception feature and the target updated perception feature until the multiple trained task processing sub-models all meet the first convergence condition, and use the multiple trained task processing sub-models as multiple target task processing sub-models.

[0096] In the disclosed embodiment, after the second perceptual feature is updated based on the target feature extraction sub-model to obtain the second updated perceptual feature, and the target perceptual feature is updated based on the target feature extraction sub-model to obtain the target updated perceptual feature, multiple initial task processing sub-models can be iteratively trained according to the second updated perceptual feature and the target updated perceptual feature, respectively, until the multiple trained task processing sub-models all meet the first convergence condition, and the multiple trained task processing sub-models are respectively used as multiple target task processing sub-models.

[0097] S212: Generate a target perception model based on the target feature extraction sub-model and multiple target task processing sub-models.

[0098] In the embodiment of the present disclosure, when the feature extraction sub-model obtained through training satisfies the second convergence condition, and the task processing sub-models obtained through training satisfy the first convergence condition, a target perception model can be generated according to the target feature extraction sub-model and multiple target task processing sub-models.

[0099] That is to say, in the embodiment of the present disclosure, the target feature extraction sub-model and multiple target task processing sub-models can be connected based on the connection layer to generate a target perception model.

[0100] In the embodiment of the present disclosure, first sample perception data collected by the vehicle is obtained, wherein the first sample perception data has a corresponding first perception feature, and second sample perception data related to the critical path information is determined, wherein the second sample perception data has a corresponding second perception feature, and then the feature similarity between the first perception feature and the second perception feature is determined, and then a plurality of task processing score values ​​related to the first perception feature are obtained, wherein the task processing score value is used for the path planning sub-model to plan the path, so that the task processing score value can be accurately obtained, and then according to the feature similarity and / or the plurality of task processing score values, it is determined that the first sample perception data is the target sample perception data, and the target sample perception data is input into the initial feature extraction sub-model to obtain the target perception feature corresponding to the target sample perception data output by the initial feature extraction sub-model, and then according to the second perception feature and the target perception feature of the second sample perception data, at least one iterative training is performed on the plurality of initial task processing sub-models respectively, until the number of iterative training is less than the number threshold, and the plurality of task processing sub-models obtained by the training all meet the first convergence condition, and the plurality of task processing sub-models obtained by the training are respectively used as the plurality of target task processing sub-models, and then according to the initial feature extraction sub-model, the plurality of task processing sub-models obtained by the training are respectively used as the plurality of target task processing sub-models. Extract submodels and multiple target task processing submodels to generate a target perception model, and when the number of iterative training reaches the number threshold and at least one trained task processing submodel does not meet the first convergence condition, perform at least one iterative training on the initial feature extraction submodel according to the second sample perception data and the target sample perception data until the trained feature extraction submodel meets the second convergence condition, use the trained feature extraction submodel as the target feature extraction submodel, and then update the second perception feature based on the target feature extraction submodel to obtain a second updated perception feature, and update the target perception feature based on the target feature extraction submodel to obtain a target updated perception feature, and perform iterative training on multiple initial task processing submodels according to the second updated perception feature and the target updated perception feature, respectively, until multiple trained task processing submodels all meet the first convergence condition, use the multiple trained task processing submodels as multiple target task processing submodels, and then generate a target perception model according to the target feature extraction submodel and the multiple target task processing submodels, so as to give full play to the data value of the collected sample perception data and effectively improve the training effect of the vehicle-side perception model based on the sample perception data.

[0101] Figure 3 is a flowchart of a perception method based on a perception model according to some embodiments of the present disclosure, such as Figure 3As shown, the perception method based on the perception model includes the following steps:

[0102] S301: Acquire the data to be sensed collected by the vehicle.

[0103] Among them, the data collected by the vehicle and currently to be used for the vehicle-side perception task is the data to be perceived. The data to be perceived can be, for example, image data collected by the vehicle-side camera, lidar data collected by the vehicle-side lidar, millimeter-wave radar data collected by the vehicle-side millimeter-wave radar, IMU data collected by the vehicle-side inertial measurement unit (IMU), etc., without any restriction.

[0104] That is to say, in the embodiments of the present disclosure, to obtain the data to be sensed collected by the vehicle, the vehicle may be pre-set with corresponding data acquisition devices, such as cameras, lidars, and IMUs, and then, during the vehicle driving process, the camera may be controlled to collect image data in the vehicle driving scene, and / or the lidar may be controlled to collect lidar data in the vehicle driving scene, and / or the IMU may be controlled to collect IMU data in the vehicle driving scene, and the aforementioned collected image data in the vehicle driving scene, and / or lidar data, and / or IMU data may be used together as the data to be sensed, and there is no limitation on this.

[0105] S302: Input the data to be sensed into the target perception model trained by the perception model training method to obtain the target perception result output by the target perception model.

[0106] Among them, the perception model obtained by the perception model training in the aforementioned embodiments of the present disclosure is a target perception model, and the target perception model can be, for example, a model for performing one or more vehicle assisted driving tasks such as target detection tasks, semantic segmentation tasks, three-dimensional perception tasks, and path planning tasks, without limitation.

[0107] That is to say, in the embodiments of the present disclosure, the data to be perceived collected by the vehicle side can be input into the target perception model, and the target perception model processes the data to be perceived to obtain a corresponding model processing result, which is the target perception result. The target perception result can be used to assist the vehicle in automatic driving.

[0108] In the disclosed embodiment, by acquiring the data to be sensed collected by the vehicle and inputting the data to be sensed into the target perception model trained by the perception model training method, so as to obtain the target perception result output by the target perception model, the target perception model can be combined to improve the execution efficiency of the vehicle-side perception task, and the vehicle can be accurately controlled based on the target perception result of the target perception model.

[0109] Figure 4 is a block diagram of a training device for a perception model according to some embodiments of the present disclosure. Figure 4 The training device 40 of the perception model includes: a first acquisition module 401, a first determination module 402, a second determination module 403 and a generation module 404, wherein:

[0110] A first acquisition module 401 is used to acquire first sample perception data collected by the vehicle, wherein the first sample perception data has a corresponding first perception feature;

[0111] A first determining module 402 is used to determine second sample perception data related to the critical path information, wherein the second sample perception data has a corresponding second perception feature;

[0112] A second determination module 403 is used to determine, based on the first perception feature and the second perception feature, that the first sample perception data is the target sample perception data, wherein a weight value of the target sample perception data in the target scene is greater than a weight threshold;

[0113] The training module 404 is used to train the initial perception model according to the second sample perception data and the target sample perception data to obtain the target perception model.

[0114] In some embodiments of the present disclosure, the initial perception model includes: a path planning sub-model; wherein the first determination module 402 is further used to:

[0115] Determine the sample perception data used when the path planning sub-model predicts the key path information, and use the used sample perception data as the second sample perception data.

[0116] In some embodiments of the present disclosure, the second determining module 403 is further configured to:

[0117] determining a feature similarity between the first perceptual feature and the second perceptual feature; and / or,

[0118] Acquire multiple task processing score values ​​related to the first perception feature, wherein the task processing score values ​​are used for the path planning sub-model to plan the path;

[0119] According to the feature similarity and / or the multiple task processing score values, it is determined that the first sample perception data is the target sample perception data.

[0120] In some embodiments of the present disclosure, the initial perception model further includes: a plurality of initial task processing sub-models; wherein the second determination module 403 is further used to:

[0121] The first perception feature is input into each initial task processing sub-model to obtain a plurality of task processing score values ​​outputted by the plurality of initial task processing sub-models respectively.

[0122] In some embodiments of the present disclosure, the second determining module 403 is further configured to:

[0123] Determine whether the feature similarity is less than a similarity threshold to obtain a first result; and / or,

[0124] Determine a scoring weight value corresponding to each task processing scoring value, perform weighted average processing on multiple task processing scoring values ​​according to the scoring weight value, and determine whether the value obtained by weighted processing is less than the task scoring threshold value to obtain a second result;

[0125] According to the first result and / or the second result, it is determined that the first sample perception data is the target sample perception data.

[0126] In some embodiments of the present disclosure, the second determining module 403 is further configured to:

[0127] If the first result is that the feature similarity is less than the similarity threshold, and / or the second result is that the value obtained by weighted processing is less than the task score threshold, it is determined that the first sample perception data is the target sample perception data.

[0128] In some embodiments of the present disclosure, the initial perception model includes: an initial feature extraction sub-model, a plurality of initial task processing sub-models;

[0129] The training module 404 is also used for:

[0130] Inputting the target sample perception data into the initial feature extraction sub-model to obtain the target perception features output by the initial feature extraction sub-model corresponding to the target sample perception data;

[0131] Performing at least one iterative training on each of the multiple initial task processing sub-models according to the second perception feature and the target perception feature of the second sample perception data, until the number of iterative trainings is less than a number threshold, and the multiple task processing sub-models obtained by training all meet the first convergence condition, and using the multiple task processing sub-models obtained by training as multiple target task processing sub-models;

[0132] A target perception model is generated according to an initial feature extraction sub-model and multiple target task processing sub-models.

[0133] In some embodiments of the present disclosure, the training module 404 is further configured to:

[0134] If the number of iterative training reaches the number threshold, and at least one task processing sub-model obtained through training does not meet the first convergence condition, the initial feature extraction sub-model is iteratively trained at least once according to the second sample perception data and the target sample perception data until the feature extraction sub-model obtained through training meets the second convergence condition, and the feature extraction sub-model obtained through training is used as the target feature extraction sub-model;

[0135] The second perceptual feature is updated based on the target feature extraction sub-model to obtain a second updated perceptual feature, and the target perceptual feature is updated based on the target feature extraction sub-model to obtain a target updated perceptual feature;

[0136] Iteratively training the multiple initial task processing sub-models according to the second updated perception feature and the target updated perception feature until the multiple trained task processing sub-models all meet the first convergence condition, and using the multiple trained task processing sub-models as the multiple target task processing sub-models;

[0137] A target perception model is generated based on the target feature extraction sub-model and multiple target task processing sub-models.

[0138] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0139] In this embodiment, by acquiring first sample perception data collected by the vehicle, wherein the first sample perception data has a corresponding first perception feature, and determining second sample perception data related to the critical path information, wherein the second sample perception data has a corresponding second perception feature, and then determining that the first sample perception data is target sample perception data based on the first perception feature and the second perception feature, wherein a weight value of the target sample perception data in the target scene is greater than a weight threshold, and then training an initial perception model based on the second sample perception data and the target sample perception data to obtain a target perception model, thereby being able to give full play to the data value of the collected sample perception data and being able to effectively improve the training effect of the vehicle-side perception model based on the sample perception data.

[0140] Figure 5 is a block diagram of a perception device based on a perception model according to some embodiments of the present disclosure. Figure 5 The perception device 50 based on the perception model includes: a second acquisition module 501 and a processing module 502, wherein:

[0141] The second acquisition module 501 is used to acquire the data to be sensed collected by the vehicle;

[0142] The processing module 502 is used to input the data to be sensed into the target perception model trained by the perception model training device to obtain the target perception result output by the target perception model.

[0143] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0144] In this embodiment, the data to be perceived collected by the vehicle is obtained, and the data to be perceived is input into the target perception model trained by the perception model training method to obtain the target perception result output by the target perception model, so that the target perception model can be combined to improve the execution efficiency of the vehicle-side perception task, and the vehicle can be accurately controlled based on the target perception result of the target perception model.

[0145] Figure 6 600 is a block diagram of a vehicle according to an exemplary embodiment. For example, vehicle 600 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. Vehicle 600 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0146] Reference Figure 6 , the vehicle 600 may include various subsystems, for example, an infotainment system 610, a perception system 620, a decision control system 630, a drive system 640, and a computing platform 650. The vehicle 600 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of the vehicle 600 may be interconnected by wire or wireless means.

[0147] In some embodiments, the infotainment system 610 may include a communication system, an entertainment system, and a navigation system, among others.

[0148] The perception system 620 may include several sensors for sensing information about the environment around the vehicle 600. For example, the perception system 620 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0149] The decision control system 630 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0150] The drive system 640 may include components that provide powered motion for the vehicle 600. In one embodiment, the drive system 640 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of an internal combustion engine, an electric motor, and an air compression engine. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0151] Some or all functions of the vehicle 600 are controlled by a computing platform 650. The computing platform 650 may include at least one processor 651 and a memory 652, and the processor 651 may execute instructions 653 stored in the memory 652.

[0152] The processor 651 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0153] The memory 652 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0154] In addition to the instructions 653 , the memory 652 may also store data, such as road maps, route information, and data such as the location, direction, and speed of the vehicle. The data stored in the memory 652 may be used by the computing platform 650 .

[0155] In the embodiment of the present disclosure, the processor 651 may execute instruction 653 to complete all or part of the steps of the above-mentioned perceptual model training method.

[0156] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the training method of the perception model provided by the present disclosure.

[0157] In addition, the word "exemplary" is used herein to indicate serving as an example, instance, or diagram. Any aspect or design described as "exemplary" in this article is not necessarily understood to be advantageous compared to other aspects or designs. On the contrary, the use of the word exemplary is intended to present concepts in a specific way. As used herein, the term "or" is intended to represent an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to represent any one of the natural inclusive arrangements. That is, if X applies A; X applies B; or X applies both A and B, "X applies A or B" is satisfied under any of the aforementioned examples. In addition, unless otherwise specified or clearly pointed to a singular form from the context, the articles "one" and "an" as used in this application and the appended claims are generally understood to mean "one or more".

[0158] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art after reading and understanding the specification and drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components (e.g., elements, resources, etc.) described above, unless otherwise noted, the terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific functions of the described components, even if the structure is not equivalent to the disclosed structure. In addition, although the specific features of the present disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be desired and conducive to any given or specific application. In addition, with respect to "including", "having", "having", "having", or variations thereof used in the specific embodiments or claims, such terms are intended to be inclusive in a manner similar to the term "comprising".

[0159] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0160] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a perception model, characterized in that: The method comprises: Acquire first sample perception data collected by the vehicle, wherein the first sample perception data has a corresponding first perception feature; Determining second sample perception data related to the critical path information, wherein the second sample perception data has a corresponding second perception feature; Determining, according to the first perception feature and the second perception feature, that the first sample perception data is target sample perception data, wherein a weight value of the target sample perception data in a target scene is greater than a weight threshold; An initial perception model is trained according to the second sample perception data and the target sample perception data to obtain a target perception model.

2. The method according to claim 1, characterized in that The initial perception model includes: a path planning sub-model; wherein the determining of the second sample perception data related to the critical path information includes: Determine the sample perception data used by the path planning sub-model to predict the key path information, and use the used sample perception data as the second sample perception data.

3. The method according to claim 2, characterized in that The determining, according to the first perception feature and the second perception feature, that the first sample perception data is target sample perception data includes: determining a feature similarity between the first perceptual feature and the second perceptual feature; and / or, Acquire a plurality of task processing score values ​​related to the first perception feature, wherein the task processing score values ​​are used by the path planning sub-model to plan a path; According to the feature similarity and / or the multiple task processing score values, it is determined that the first sample perception data is the target sample perception data.

4. The method according to claim 3, characterized in that The initial perception model further includes: a plurality of initial task processing sub-models; wherein the obtaining of a plurality of task processing score values ​​related to the first perception feature includes: The first perception feature is input into each of the initial task processing sub-models to obtain the multiple task processing score values ​​output by the multiple initial task processing sub-models respectively.

5. The method according to claim 3 or 4, characterized in that The determining, according to the feature similarity and / or the multiple task processing score values, that the first sample perception data is target sample perception data includes: Determine whether the feature similarity is less than a similarity threshold to obtain a first result; and / or, Determine a scoring weight value corresponding to each of the task processing scoring values, perform weighted averaging processing on the multiple task processing scoring values ​​according to the scoring weight value, and determine whether the value obtained by the weighted processing is less than the task scoring threshold value, so as to obtain a second result; According to the first result and / or the second result, it is determined that the first sample perception data is the target sample perception data.

6. The method according to claim 5, characterized in that The determining, according to the first result and / or the second result, that the first sample perception data is the target sample perception data includes: If the first result is that the feature similarity is less than the similarity threshold, and / or the second result is that the value obtained by the weighted processing is less than the task score threshold, it is determined that the first sample perception data is the target sample perception data.

7. The method according to any one of claims 1 to 6, characterized in that: The initial perception model includes: an initial feature extraction sub-model and a plurality of initial task processing sub-models; The step of training the initial perception model according to the second sample perception data and the target sample perception data to obtain the target perception model includes: Inputting the target sample perception data into the initial feature extraction sub-model to obtain the target perception features output by the initial feature extraction sub-model and corresponding to the target sample perception data; According to the second perception feature of the second sample perception data and the target perception feature, the multiple initial task processing sub-models are respectively iteratively trained for at least one time until the number of iterative trainings is less than a number threshold, and the multiple task processing sub-models obtained by training all meet the first convergence condition, and the multiple task processing sub-models obtained by training are respectively used as multiple target task processing sub-models; The target perception model is generated according to the initial feature extraction sub-model and the multiple target task processing sub-models.

8. The method according to claim 7, characterized in that The training of the initial perception model according to the second sample perception data and the target sample perception data to obtain a target perception model further includes: If the number of iterative training reaches the number threshold, and at least one of the task processing sub-models obtained through training does not meet the first convergence condition, the initial feature extraction sub-model is iteratively trained at least once according to the second sample perception data and the target sample perception data until the feature extraction sub-model obtained through training meets the second convergence condition, and the feature extraction sub-model obtained through training is used as the target feature extraction sub-model; updating the second perceptual feature based on the target feature extraction submodel to obtain a second updated perceptual feature, and updating the target perceptual feature based on the target feature extraction submodel to obtain a target updated perceptual feature; Iteratively training the multiple initial task processing sub-models according to the second update perception feature and the target update perception feature until the multiple task processing sub-models obtained by training all meet the first convergence condition, and using the multiple task processing sub-models obtained by training as the multiple target task processing sub-models; The target perception model is generated according to the target feature extraction sub-model and the multiple target task processing sub-models.

9. A perception method based on a perception model, characterized in that: The method comprises: Obtain the data to be sensed collected by the vehicle; The data to be sensed is input into a target perception model trained by the perception model training method as described in any one of claims 1 to 8 above, so as to obtain a target perception result output by the target perception model.

10. A training device for a perception model, characterized in that: The device comprises: A first acquisition module, configured to acquire first sample perception data collected by the vehicle, wherein the first sample perception data has a corresponding first perception feature; A first determining module, configured to determine second sample perception data related to the critical path information, wherein the second sample perception data has a corresponding second perception feature; a second determination module, configured to determine, based on the first perception feature and the second perception feature, that the first sample perception data is target sample perception data, wherein a weight value of the target sample perception data in a target scene is greater than a weight threshold; A training module is used to train an initial perception model according to the second sample perception data and the target sample perception data to obtain a target perception model.

11. A perception device based on a perception model, characterized in that: The device comprises: The second acquisition module is used to acquire the data to be sensed collected by the vehicle; A processing module is used to input the data to be sensed into the target perception model trained by the perception model training device as described in claim 10 above, so as to obtain the target perception result output by the target perception model.

12. A vehicle, characterized in that: include: processor; A memory for storing processor executable instructions; wherein the processor is configured to: implement the steps of the method described in any one of claims 1-8, or implement the steps of the method described in claim 9.

13. A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by the processor of a mobile terminal, enables the mobile terminal to execute the training method of the perception model described in any one of claims 1-8, or implement the perception method based on the perception model described in claim 9.

Images