Method, device, storage medium and equipment for training and using perception model
By training the CGAN model and the recognition model, the applicability of the perception model across different sensor systems was solved, achieving universality under different acquisition angles and cross-vehicle application.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANKAISI INTELLIGENT TECH CO LTD GUIZHOU
- Filing Date
- 2023-09-05
- Publication Date
- 2026-05-29
AI Technical Summary
Existing perception models are not universally applicable across different sensor systems, especially when the sensor's acquisition angle changes, leading to performance degradation. Furthermore, the sensor systems of different automakers vary greatly, making cross-automaker applications difficult.
A conditional generative adversarial network (CGAN) model and a recognition model are used. By training sensor data and angle deviation angles in the training samples, the generator and discriminator are trained. The generator can generate sensor data corresponding to different acquisition angles, and the recognition model can identify environmental objects.
This achieves the universality of the perception model across sensor systems with different acquisition angles, improving the applicability of the perception model and its cross-vehicle application capabilities.
Smart Images
Figure CN117152705B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, and in particular to a method, apparatus, storage medium, and device for training and using a perception model. Background Technology
[0002] Autonomous driving systems collect sensor data through various sensors installed on the vehicle body and analyze the sensor data through pre-trained perception models to obtain semantic information about the surrounding environment, thereby providing a basis for the planning and control of autonomous vehicles.
[0003] However, existing perception models can only analyze sensor data whose distribution closely resembles that of their training data. If the sensor's acquisition angle changes, causing the acquired sensor data to differ from the training data distribution, the perception model's performance will degrade. The structure of sensor systems may vary between different vehicle models, and this structure may change over time due to design upgrades. Therefore, the effectiveness of perception models trained for specific sensor systems will decrease. Furthermore, since autonomous vehicles developed by different automakers are typically equipped with different sensor systems, a perception model trained by one automaker is difficult to apply to another, which will become an obstacle to the further expansion and development of the autonomous driving community.
[0004] Although many researchers have tried to improve perception models by focusing on their generalization ability, there is no particularly effective way to enhance the versatility of perception models because neural networks are black boxes. Summary of the Invention
[0005] This application provides a method, apparatus, storage medium, and device for training and using a perception model, to address the problem that perception models cannot be universally applied across sensor systems with different acquisition angles. The technical solution is as follows:
[0006] On the one hand, a method for training a perception model is provided, the method comprising:
[0007] Create a perception model, which includes a conditional generative adversarial network (CGAN) model and a recognition model, wherein the CGAN model includes a generator and a discriminator;
[0008] Multiple sets of training samples are acquired. The training samples include three sets of sensor data collected simultaneously by three sensors in the sensor combination, a first position deviation angle between two outer sensors, and a second position deviation angle between the middle sensor and an outer sensor at a predetermined orientation. The sensor combination includes multiple sensors of the same type installed around the body of the autonomous vehicle. The sensor data is used to represent the environmental image around the vehicle.
[0009] For each training sample, the two sets of sensor data collected by the two outer sensors, the first position deviation angle and the second position deviation angle are used as the input of the generator, and the output of the generator and a set of sensor data collected by the middle sensor are used as the input of the discriminator to train the CGAN model.
[0010] The recognition model is trained.
[0011] In one possible implementation, training the recognition model includes:
[0012] During the training of the generator, the latent space generated by the generator is obtained, which is a high-dimensional representation space of the input content of the generator.
[0013] Obtain annotation information of a set of sensor data collected by intermediate sensors. The annotation information is used to identify each object in the environmental image corresponding to the sensor data.
[0014] The recognition model is trained based on the latent space and the annotation information.
[0015] In one possible implementation, training the recognition model includes:
[0016] During the training of the generator, the output content generated by the generator is obtained;
[0017] Obtain annotation information of a set of sensor data collected by intermediate sensors. The annotation information is used to identify each object in the environmental image corresponding to the sensor data.
[0018] The recognition model is trained based on the output content and the annotation information.
[0019] In one possible implementation, obtaining multiple sets of training samples includes:
[0020] Choose any three sensors from the sensor combination;
[0021] For the three sensors, the first position deviation angle and the second position deviation angle are determined based on the position information of each sensor;
[0022] Acquire sensor data collected by the three sensors at the same time;
[0023] The three sets of sensor data, the first position deviation angle, and the second position deviation angle are combined to form a training sample.
[0024] In one possible implementation, the sensor is a conventional camera or a depth camera, and the sensor data is image data; or, the sensor is a lidar, and the sensor data is point cloud data.
[0025] On the one hand, a method for using a perception model is provided for use in a perception model trained as described above, the method comprising:
[0026] Acquire two sets of sensor data collected simultaneously by the two outer sensors, as well as the first position deviation angle between the two outer sensors;
[0027] Obtain the second position deviation angle, which represents the position deviation angle between the estimated sensor data acquisition position and an outer sensor at a predetermined orientation;
[0028] The generator is used to process the two sets of sensor data, the first position deviation angle, and the second position deviation angle to obtain the generation result;
[0029] The generated result is identified using the recognition model to obtain a perception result, which represents the object in the environmental image corresponding to the acquisition location.
[0030] In one possible implementation, when the processing result is the latent space, the step of using the recognition model to recognize the generated result to obtain a perception result includes:
[0031] The latent space is identified using the aforementioned recognition model to obtain the perception result.
[0032] In one possible implementation, when the processing result is the output content, the step of using the recognition model to recognize the generated result and obtain a perception result includes:
[0033] The output content is identified using the recognition model to obtain a perception result.
[0034] On the one hand, a training device for a perception model is provided, the device comprising:
[0035] A creation module is used to create a perception model, which includes a conditional generative adversarial network (CGAN) model and a recognition model, wherein the CGAN model includes a generator and a discriminator.
[0036] The acquisition module is used to acquire multiple sets of training samples. The training samples include three sets of sensor data collected simultaneously by three sensors in the sensor combination, a first position deviation angle between two outer sensors, and a second position deviation angle between the middle sensor and an outer sensor at a predetermined orientation. The sensor combination includes multiple sensors of the same type installed around the body of the autonomous vehicle. The sensor data is used to represent the environmental image around the vehicle body.
[0037] The training module is used to train the CGAN model for each training sample by taking two sets of sensor data collected by the two outer sensors, the first position deviation angle and the second position deviation angle as the input of the generator, and taking the output of the generator and a set of sensor data collected by the middle sensor as the input of the discriminator.
[0038] The training module is also used to train the recognition model.
[0039] On the one hand, a perception model is provided, which is trained by the method described above, and the perception model includes a generator and a recognition model;
[0040] The generator is used to acquire two sets of sensor data collected simultaneously by the two outer sensors and the first position deviation angle between the two outer sensors.
[0041] Obtain the second position deviation angle, which represents the position deviation angle between the estimated sensor data acquisition position and an outer sensor at a predetermined orientation;
[0042] The generator is also used to process the two sets of sensor data, the first position deviation angle, and the second position deviation angle to obtain the generation result;
[0043] The recognition model is used to identify the generated result to obtain a perception result, which represents the object in the environmental image corresponding to the acquisition location.
[0044] On one hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the training method of the perception model as described above, or the at least one instruction being loaded and executed by a processor to implement the usage method of the perception model as described above.
[0045] On one hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to implement the training method of the perception model as described above, or the instruction being loaded and executed by the processor to implement the usage method of the perception model as described above.
[0046] The beneficial effects of the technical solution provided in this application include at least the following:
[0047] When training the CGAN model, the first positional deviation angle between the two outer sensors and the second positional deviation angle between the intermediate sensor and an outer sensor at a predetermined orientation are used as constraints. The generator and discriminator in the CGAN model are trained using three sets of sensor data to enable the generator to recognize sensor data corresponding to the acquisition angle of the intermediate sensor. Then, when using the CGAN model, the first and second positional deviation angles, along with the sensor data from the two outer sensors, are input into the generator. This allows the generation of sensor data with the acquisition angle corresponding to the second positional deviation angle. Thus, the perception model can generate sensor data with the required acquisition angle, making it universally applicable across sensor systems with different acquisition angles. Attached Figure Description
[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 This is a flowchart of a method for training a perception model according to an embodiment of this application;
[0050] Figure 2 This is a schematic diagram of the sensor distribution provided in one embodiment of this application;
[0051] Figure 3 This is a schematic diagram illustrating the selection of a sensor according to an embodiment of this application;
[0052] Figure 4 This is a flowchart of a method for training a perception model according to another embodiment of this application;
[0053] Figure 5 This is a structural block diagram of a training device for a perception model provided in another embodiment of this application;
[0054] Figure 6 This is a structural block diagram of a perception model provided in another embodiment of this application. Detailed Implementation
[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0056] Different autonomous vehicles may have sensor systems that collect data from different angles. For example, one autonomous vehicle might have four sensors installed at the front, rear, left, and right of the vehicle, while another autonomous vehicle might have four sensors installed at the front left, rear left, front right, and rear right. These two sensor systems have different collection angles. The sensors mentioned here can be ordinary cameras, depth cameras, LiDAR, etc.
[0057] Existing perception models can only analyze sensor data whose distribution is similar to that of their training data. If the sensor's acquisition angle changes, causing the acquired sensor data to have a different distribution from the training data, the performance of the perception model will degrade. To enable the perception model to be universal across sensor systems with different acquisition angles, we propose a new perception model that can achieve angle transformation of sensor data between different autonomous vehicles.
[0058] Please refer to Figure 1 This document illustrates a flowchart of a method for training a perception model according to an embodiment of this application. This method can be applied to a computer device. The training method for the perception model may include:
[0059] Step 101: Create a perception model, which includes a CGAN model and a recognition model. The CGAN model includes a generator and a discriminator.
[0060] The CGAN model stands for Conditional Generative Adversarial Network, which consists of a generator and a discriminator. During training, the generator G aims to generate realistic images to deceive the discriminator D; the discriminator D aims to distinguish the images generated by the generator G from real images. Thus, the generator G and the discriminator D constitute a dynamic game.
[0061] The recognition model can be a classifier that can identify objects in the environment surrounding an autonomous vehicle.
[0062] Step 102: Obtain multiple sets of training samples. The training samples include three sets of sensor data collected simultaneously by the three sensors in the sensor combination, the first position deviation angle between the two outer sensors, and the second position deviation angle between the middle sensor and an outer sensor at a predetermined orientation. The sensor combination includes multiple sensors of the same type installed around the body of the autonomous vehicle. The sensor data is used to represent the environmental image around the vehicle body.
[0063] Autonomous vehicles are equipped with multiple sensors of the same type, and the sensor data collected by these sensors can represent the environment around the vehicle. For example, the sensors may be ordinary cameras or depth cameras, and the sensor data may be image data; or the sensors may be LiDAR, and the sensor data may be point cloud data.
[0064] Specifically, obtaining multiple sets of training samples can include the following sub-steps:
[0065] (1) Select any three sensors from the sensor combination.
[0066] Assume that an autonomous vehicle is equipped with 9 sensors, and the locations of these 9 sensors are as follows: Figure 2 As shown, you can choose from three sensors: 0°, 4° and 74°; or three sensors: 0°, 4° and 120°; or three sensors: 0°, 74° and 120°; or three sensors: 0°, 157° and 240°; or three sensors: 120°, 213° and 304°, and so on.
[0067] like Figure 3 As shown, Cam Pos indicates the camera position, and each row of symbols represents the three selected sensors, with "√" indicating the outer sensor and "○" indicating the middle sensor. The last three rows of "√" symbols correspond to 0°, representing 360°.
[0068] (2) For the three sensors, determine the first position deviation angle and the second position deviation angle based on the position information of each sensor.
[0069] The first position deviation angle represents the position deviation angle between the two outer sensors, and the second position deviation angle represents the position deviation angle between the middle sensor and an outer sensor at a predetermined orientation. The predetermined orientation can be either the left or right side, and can be set according to business requirements.
[0070] For each of the three selected sensors, the first position deviation angle and the second position deviation angle corresponding to these three sensors can be calculated.
[0071] Assuming that the three sensors selected are 0°, 4° and 74°, and the predetermined orientation is to the left, then the first position deviation angle = 74 - 0 = 74°, and the second position deviation angle = 4 - 0 = 4°.
[0072] Assuming that the three sensors selected are 120°, 213° and 304°, and the predetermined orientation is to the left, then the first position deviation angle = 304 - 120 = 184°, and the second position deviation angle = 213 - 120 = 93°.
[0073] (3) Obtain sensor data collected by the three sensors at the same time.
[0074] In this embodiment, sensor data collected by three sensors can be acquired multiple times at the same time to obtain multiple sets of sensor data.
[0075] (4) Combine the three sets of sensor data, the first position deviation angle, and the second position deviation angle into a training sample.
[0076] In this embodiment, each set of sensor data can be combined with the corresponding first position deviation angle and second position deviation angle to form a set of training samples.
[0077] Step 103: For each training sample, the two sets of sensor data collected by the two outer sensors, the first position deviation angle and the second position deviation angle are used as the input of the generator, and the output of the generator and a set of sensor data collected by the middle sensor are used as the input of the discriminator to train the CGAN model.
[0078] For each training sample, the first position deviation angle and the second position deviation angle can be used as constraints, and together with the two sets of sensor data collected by the two outer sensors, they can be used as the input to the generator. The generator generates the estimated sensor data corresponding to the second position deviation angle based on the input. The estimated sensor data and the actual sensor data collected by the intermediate sensor are used as the input to the discriminator. The discriminator judges whether the estimated sensor data is the same as the actual sensor data, so as to train the generator and the discriminator.
[0079] Step 104: Train the recognition model.
[0080] In this embodiment, the recognition model has two types of inputs: one is the latent space (intermediate data) generated by the generator, which is a high-dimensional representation space of the generator's input content; the other is the generator's output content.
[0081] (1) When the input is the latent space, training the recognition model may include: during the training of the generator, obtaining the latent space generated by the generator, which is a high-dimensional representation space of the generator's input content; obtaining the annotation information of a set of sensor data collected by the intermediate sensor, which is used to identify each object in the environmental picture corresponding to the sensor data; and training the recognition model based on the latent space and the annotation information.
[0082] In this embodiment, the generator first downsamples the input content, then uses a residual network to process the downsampled result as a residual, and finally upsamples the residual result to obtain the output content. When the residual network includes multiple residual modules, the latent space can be the output content of the residual module located in the middle position of the residual network. Here, the latent space can be understood as a vector.
[0083] Labeling information is used to identify objects in the environment. For example, if the objects in the environment include pedestrians, buildings, and traffic lights, the labeling information includes the bounding box and pedestrian label for pedestrians, the bounding box and building label for buildings, and the bounding box and traffic light label for traffic lights.
[0084] In this implementation, the input to the recognition model is the latent space, and the output is the bounding box and the classification label.
[0085] (2) When the input is the output content, training the recognition model may include: during the training of the generator, obtaining the output content generated by the generator; obtaining the annotation information of a set of sensor data collected by the intermediate sensor, the annotation information being used to identify each object in the environmental picture corresponding to the sensor data; and training the recognition model based on the output content and the annotation information.
[0086] In this embodiment, the generator first downsamples the input content, then uses a residual network to process the residuals of the downsampled result, and finally upsamples the residuals to obtain the output content. The output content here refers to a set of estimated sensor data, which can be an image or a frame of point cloud data.
[0087] Labeling information is used to identify objects in the environment. For example, if the objects in the environment include pedestrians, buildings, and traffic lights, then the labeling information includes the bounding box and pedestrian label for pedestrians, the bounding box and building label for buildings, and the bounding box and traffic light label for traffic lights.
[0088] In this implementation, the input to the recognition model is image or point cloud data, and the output is bounding boxes and classification labels.
[0089] In summary, the training method for the perception model provided in this application uses the first positional deviation angle between two outer sensors and the second positional deviation angle between the intermediate sensor and an outer sensor at a predetermined orientation as constraints on the CGAN model during training. The generator and discriminator in the CGAN model are trained using three sets of sensor data, enabling the generator to recognize sensor data corresponding to the acquisition angle of the intermediate sensor. Then, when using the CGAN model, the first positional deviation angle, the second positional deviation angle, and the sensor data from the two outer sensors are input into the generator to generate sensor data with the acquisition angle corresponding to the second positional deviation angle. This allows the perception model to generate sensor data with the required acquisition angle, making it universally applicable across sensor systems with different acquisition angles.
[0090] Please refer to Figure 4 This document illustrates a flowchart of a method for using a perception model according to an embodiment of this application. This method of using the perception model can be applied to a computer device. The method of using the perception model may include:
[0091] Step 401: Obtain two sets of sensor data collected simultaneously by the two outer sensors and the first position deviation angle between the two outer sensors.
[0092] The two sets of sensor data obtained by the perception model are collected by two outer sensors, and the first positional deviation angle between the two outer sensors can be calculated based on the position of the two outer sensors.
[0093] Step 402: Obtain the second position deviation angle, which represents the position deviation angle between the estimated sensor data acquisition position and an outer sensor at a predetermined orientation.
[0094] The perception model also needs to obtain the second position deviation angle. Calculating the second position deviation angle may require first obtaining the acquisition position of the simulated intermediate sensor, and then calculating the second position deviation angle between this acquisition position and the position of an outer sensor at a predetermined orientation.
[0095] Still with Figure 2 Taking the sensor combination shown as an example, assuming that the sensor data acquired by the two outer sensors at 0° and 74° is obtained, and it is necessary to simulate the sensor data acquired by the 4° sensor, and the predetermined orientation is to the left, then the first position deviation angle = 74° and the second position deviation angle = 4° can be calculated.
[0096] Step 403: Use the generator to process the two sets of sensor data, the first position deviation angle, and the second position deviation angle to obtain the generation result.
[0097] In this embodiment, the first position deviation angle and the second position deviation angle are used as conditional constraints, and together with the two sets of sensor data collected by the two outer sensors, they serve as the input content for the generator. The generator generates a result based on the input content. The generated result is either the latent space or the output content.
[0098] Step 404: Use the recognition model to identify the generated result to obtain the perception result, which represents the object in the environmental image corresponding to the acquisition location.
[0099] When the generated result is a latent space, the generated result is identified using a recognition model to obtain a perceptual result. This can include: using a recognition model to identify the latent space to obtain a perceptual result.
[0100] When the generated result is output content, the generated result is identified using a recognition model to obtain a perceived result. This can include: using a recognition model to identify the output content to obtain a perceived result.
[0101] The perception results include the bounding box of the object and the classification label.
[0102] In summary, the method for using the perception model provided in this application, when training the CGAN model, uses the first positional deviation angle between the two outer sensors and the second positional deviation angle between the intermediate sensor and an outer sensor at a predetermined orientation as constraints on the CGAN model. The generator and discriminator in the CGAN model are trained using three sets of sensor data, enabling the generator to recognize sensor data corresponding to the acquisition angle of the intermediate sensor. Then, when using the CGAN model, the first positional deviation angle, the second positional deviation angle, and the sensor data from the two outer sensors are input into the generator to generate sensor data with the acquisition angle corresponding to the second positional deviation angle. This allows the perception model to generate sensor data with the required acquisition angle, making it universally applicable across sensor systems with different acquisition angles.
[0103] Please refer to Figure 5 This diagram illustrates a structural block diagram of a training apparatus for a perception model according to an embodiment of this application. This training apparatus for the perception model can be applied to a computer device. The training apparatus for the perception model may include:
[0104] Module 510 is created to create a perception model, which includes a conditional generative adversarial network (CGAN) model and a recognition model. The CGAN model includes a generator and a discriminator.
[0105] The acquisition module 520 is used to acquire multiple sets of training samples. The training samples include three sets of sensor data collected simultaneously by three sensors in the sensor combination, a first position deviation angle between two outer sensors, and a second position deviation angle between the middle sensor and an outer sensor at a predetermined orientation. The sensor combination includes multiple sensors of the same type installed around the body of the autonomous vehicle. The sensor data is used to represent the environmental image around the vehicle body.
[0106] The training module 530 is used to train the CGAN model for each training sample by taking two sets of sensor data collected by the two outer sensors, the first position deviation angle and the second position deviation angle as the input of the generator, and taking the output of the generator and a set of sensor data collected by the middle sensor as the input of the discriminator.
[0107] Training module 530 is also used to train the recognition model.
[0108] In an optional embodiment, the training module 530 is further configured to:
[0109] During the training of the generator, the latent space generated by the generator is obtained. The latent space is a high-dimensional representation space of the generator's input content.
[0110] Obtain annotation information for a set of sensor data collected by intermediate sensors. The annotation information is used to identify various objects in the environmental image corresponding to the sensor data.
[0111] The recognition model is trained based on the latent space and annotation information.
[0112] In an optional embodiment, the training module 530 is further configured to:
[0113] During the training of the generator, obtain the output content generated by the generator;
[0114] Obtain annotation information for a set of sensor data collected by intermediate sensors. The annotation information is used to identify various objects in the environmental image corresponding to the sensor data.
[0115] The recognition model is trained based on the output content and annotation information.
[0116] In an optional embodiment, the acquisition module 520 is further configured to:
[0117] Choose any three sensors from the sensor combination;
[0118] For three sensors, the first position deviation angle and the second position deviation angle are determined based on the position information of each sensor.
[0119] Acquire sensor data collected by three sensors at the same time;
[0120] The three sets of sensor data, the first position deviation angle, and the second position deviation angle are combined to form a training sample.
[0121] In one optional embodiment, the sensor is a conventional camera or a depth camera, and the sensor data is image data; or, the sensor is a LiDAR, and the sensor data is point cloud data.
[0122] In summary, the training apparatus for the perception model provided in this application, when training the CGAN model, uses the first positional deviation angle between the two outer sensors and the second positional deviation angle between the intermediate sensor and an outer sensor at a predetermined orientation as constraints on the CGAN model. It uses three sets of sensor data to train the generator and discriminator in the CGAN model, enabling the generator to recognize sensor data corresponding to the acquisition angle of the intermediate sensor. Then, when using the CGAN model, the first positional deviation angle, the second positional deviation angle, and the sensor data from the two outer sensors are input into the generator to generate sensor data with the acquisition angle corresponding to the second positional deviation angle. This allows the perception model to generate sensor data with the required acquisition angle, making it universally applicable among sensor systems with different acquisition angles.
[0123] Please refer to Figure 6 This diagram illustrates a structural block diagram of a perception model provided in one embodiment of this application, which can be applied to a computer device. The perception model may include a generator 610 and a recognition model 620.
[0124] Generator 610 is used to acquire two sets of sensor data collected simultaneously by the two outer sensors and the first position deviation angle between the two outer sensors.
[0125] The generator 610 is also used to obtain a second position deviation angle, which represents the position deviation angle between the estimated sensor data acquisition position and an outer sensor at a predetermined orientation.
[0126] The generator 610 is also used to process the two sets of sensor data, the first position deviation angle, and the second position deviation angle to obtain the generation result;
[0127] Recognition model 620 is used to identify the generated results and obtain the perception results, which represent the objects in the environmental image corresponding to the acquisition location.
[0128] In an optional embodiment, when the generated result is a latent space, the recognition model 620 is further used for:
[0129] The latent space is identified using a recognition model to obtain the perception results.
[0130] In an optional embodiment, when the generated result is output content, the recognition model 620 is further configured to:
[0131] The output content is processed by a recognition model to obtain the perception result.
[0132] In summary, the sensing model application device provided in this application, when training the CGAN model, uses the first positional deviation angle between the two outer sensors and the second positional deviation angle between the intermediate sensor and an outer sensor at a predetermined orientation as constraints on the CGAN model. It trains the generator and discriminator in the CGAN model using three sets of sensor data, enabling the generator to recognize sensor data corresponding to the acquisition angle of the intermediate sensor. Then, when using the CGAN model, the first positional deviation angle, the second positional deviation angle, and the sensor data from the two outer sensors are input into the generator to generate sensor data with the acquisition angle corresponding to the second positional deviation angle. This allows the sensing model to generate sensor data with the required acquisition angle, making it universally applicable among sensor systems with different acquisition angles.
[0133] One embodiment of this application provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the training method of the perception model as described above, or the at least one instruction is loaded and executed by a processor to implement the usage method of the perception model as described above.
[0134] One embodiment of this application provides a computer device, the computer device including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to implement the training method of the perception model as described above, or the instruction being loaded and executed by the processor to implement the usage method of the perception model as described above.
[0135] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0136] The above description is not intended to limit the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.
Claims
1. A method for training a perceptual model, characterized in that, The method includes: Create a perception model, which includes a conditional generative adversarial network (CGAN) model and a recognition model, wherein the CGAN model includes a generator and a discriminator; Multiple sets of training samples are acquired. The training samples include three sets of sensor data collected simultaneously by three sensors in the sensor combination, a first position deviation angle between two outer sensors, and a second position deviation angle between the middle sensor and an outer sensor at a predetermined orientation. The sensor combination includes multiple sensors of the same type installed around the body of the autonomous vehicle. The sensor data is used to represent the environmental image around the vehicle. For each training sample, the two sets of sensor data collected by the two outer sensors, the first position deviation angle and the second position deviation angle are used as the input of the generator, and the output of the generator and a set of sensor data collected by the middle sensor are used as the input of the discriminator to train the CGAN model. The recognition model is trained; The step of acquiring multiple sets of training samples includes: arbitrarily selecting three sensors from the sensor combination; for the three sensors, determining the first position deviation angle and the second position deviation angle based on the position information of each sensor; acquiring sensor data collected by the three sensors at the same time; and combining the three sets of sensor data, the first position deviation angle, and the second position deviation angle into a set of training samples.
2. The training method for the perception model according to claim 1, characterized in that, Training the recognition model includes: During the training of the generator, the latent space generated by the generator is obtained, which is a high-dimensional representation space of the input content of the generator. Obtain annotation information of a set of sensor data collected by intermediate sensors. The annotation information is used to identify each object in the environmental image corresponding to the sensor data. The recognition model is trained based on the latent space and the annotation information.
3. The training method for the perception model according to claim 1, characterized in that, Training the recognition model includes: During the training of the generator, the output content generated by the generator is obtained; Obtain annotation information of a set of sensor data collected by intermediate sensors. The annotation information is used to identify each object in the environmental image corresponding to the sensor data. The recognition model is trained based on the output content and the annotation information.
4. The training method for the perceptual model according to any one of claims 1 to 3, characterized in that, The sensor is a conventional camera or a depth camera, and the sensor data is image data; or... The sensor is a lidar, and the sensor data is point cloud data.
5. A method for using a perception model, characterized in that, For use in a perception model trained according to any one of claims 1 to 4, the method comprises: Acquire two sets of sensor data collected simultaneously by the two outer sensors, as well as the first position deviation angle between the two outer sensors; Obtain the second position deviation angle, which represents the position deviation angle between the estimated sensor data acquisition position and an outer sensor at a predetermined orientation; The generator is used to process the two sets of sensor data, the first position deviation angle, and the second position deviation angle to obtain the generation result; The generated result is identified using the recognition model to obtain a perception result, which represents the object in the environmental image corresponding to the acquisition location.
6. The method of using the perception model according to claim 5, characterized in that, When the processing result is the latent space, the step of using the recognition model to recognize the generated result and obtain the perception result includes: The latent space is identified using the aforementioned recognition model to obtain the perception result.
7. The method of using the perception model according to claim 5, characterized in that, When the processing result is the output content, the step of using the recognition model to recognize the generated result and obtain a perception result includes: The output content is identified using the recognition model to obtain a perception result.
8. A training device for a perception model, characterized in that, The device includes: A creation module is used to create a perception model, which includes a conditional generative adversarial network (CGAN) model and a recognition model, wherein the CGAN model includes a generator and a discriminator. The acquisition module is used to acquire multiple sets of training samples. The training samples include three sets of sensor data collected simultaneously by three sensors in the sensor combination, a first position deviation angle between two outer sensors, and a second position deviation angle between the middle sensor and an outer sensor at a predetermined orientation. The sensor combination includes multiple sensors of the same type installed around the body of the autonomous vehicle. The sensor data is used to represent the environmental image around the vehicle body. The training module is used to train the CGAN model for each training sample by taking two sets of sensor data collected by the two outer sensors, the first position deviation angle and the second position deviation angle as the input of the generator, and taking the output of the generator and a set of sensor data collected by the middle sensor as the input of the discriminator. The training module is also used to train the recognition model; The acquisition module is further configured to: arbitrarily select three sensors from the sensor combination; for the three sensors, determine the first position deviation angle and the second position deviation angle based on the position information of each sensor; acquire sensor data collected by the three sensors at the same time; and combine the three sets of sensor data, the first position deviation angle, and the second position deviation angle into a training sample.
9. A perception model, characterized in that, The perception model is trained by any one of claims 1 to 4, and the perception model includes a generator and a recognition model; The generator is used to acquire two sets of sensor data collected simultaneously by the two outer sensors and the first position deviation angle between the two outer sensors. Obtain the second position deviation angle, which represents the position deviation angle between the estimated sensor data acquisition position and an outer sensor at a predetermined orientation; The generator is also used to process the two sets of sensor data, the first position deviation angle, and the second position deviation angle to obtain the generation result; The recognition model is used to identify the generated result to obtain a perception result, which represents the object in the environmental image corresponding to the acquisition location.
10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the training method of the perceptual model as described in any one of claims 1 to 4, or the at least one instruction is loaded and executed by a processor to implement the method of using the perceptual model as described in any one of claims 5 to 7.
11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to implement a training method for the perceptual model as described in any one of claims 1 to 4, or the instruction being loaded and executed by the processor to implement a method of using the perceptual model as described in any one of claims 5 to 7.