Multi-task model training method, image processing method, device and equipment

Abstract

The invention discloses a multi-task model training method, an image processing method, a device and equipment. The method comprises the following steps: acquiring a plurality of initial teacher models in one-to-one correspondence with a plurality of image reasoning tasks, and an initial student model for executing the plurality of image reasoning tasks; the image reasoning task is a reasoning task based on a plurality of environment images of the vehicle; the number of channels of the images processed by each initial teacher model is the same, and the sum of the number of channels of the images processed by the plurality of initial teacher models is the number of channels of the images processed by the initial student model; based on a preset environment image sample set, determining a first image feature of each initial teacher model and a second image feature of the initial student model; and training the initial student model based on the first image features corresponding to the plurality of initial teacher models and the second image features of the initial student model to obtain a trained student model.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and specifically to a multi-task model training method, image processing method, device and equipment. Background Art

[0002] With the rapid development of autonomous driving technology, Bird's Eye View (BEV) has become one of the key technologies in autonomous driving systems. BEV perception generates a unified bird's-eye view image by integrating data from multiple sensors. BEV perception enables autonomous driving systems to understand the environment around the vehicle more intuitively and comprehensively, thereby greatly improving perception accuracy and decision-making capabilities.

[0003] In autonomous driving systems, multi-task training is a common strategy, but in BEV multi-task training, there may be conflicts between different tasks. For example, the object detection task requires high-precision bounding box prediction, while the semantic segmentation task requires fine pixel-level classification.

[0004] Since the BEV representation is an intermediate representation shared by multiple tasks, different tasks may have different or even contradictory requirements for BEV representation. These conflicts not only affect the performance of the model, but also impose strict requirements on task priority and resource allocation. If there is no additional information guidance during the training process, it is easy to cause the model performance to deteriorate. Therefore, there is an urgent need for a training method that can be applied to BEV multi-task training. Summary of the invention

[0005] In view of this, the embodiments of the present application at least provide a multi-task model training method, image processing method, device and equipment.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a training method for a multi-task model, the training method comprising:

[0008] Acquire multiple initial teacher models corresponding to multiple image reasoning tasks, and an initial student model for performing the multiple image reasoning tasks; the image reasoning tasks are reasoning tasks based on multiple environmental images of the vehicle; the number of channels of the images processed by each initial teacher model is the same, and the sum of the number of channels of the images processed by the multiple initial teacher models is the number of channels of the images processed by the initial student model;

[0009] Based on a preset set of environmental image samples, determining a first image feature of each initial teacher model and a second image feature of an initial student model;

[0010] Based on the first image features corresponding to the multiple initial teacher models and the second image features of the initial student model, the initial student model is trained to obtain a trained student model.

[0011] In an embodiment of the present application, an initial teacher model and an initial student model corresponding to different image reasoning tasks are first constructed; wherein the number of channels of the image processed by each initial teacher model is the same, and the sum of the number of channels of the images processed by multiple initial teacher models is the number of channels of the image processed by the initial student model. Then the first image feature of each initial teacher model and the second image feature of the initial student model are determined. Finally, based on the first image features corresponding to the multiple initial teacher models and the second image features of the initial student model, the initial student model is trained to obtain a trained student model. In this way, by training the initial student model with the first image features output by different initial teacher models, the initial student model can learn the feature distributions of different initial teacher models, thereby enabling a single student model to learn the differentiation between different tasks from multiple teacher models, and further enabling the trained student model to perform multiple image reasoning tasks.

[0012] In some embodiments, each first image feature includes a first bird's-eye view image feature of an initial teacher model and a first feature map of the initial teacher model; the second image feature includes a second bird's-eye view image feature of an initial student model and a second feature map of the initial student model; based on the first image features corresponding to multiple initial teacher models and the second image features of the initial student model, the initial student model is trained, including at least one of the following: based on multiple first bird's-eye view image features and second bird's-eye view image features, the initial student model is trained with first distillation training; based on multiple first feature maps and second feature maps, the initial student model is trained with second distillation training; based on multiple first bird's-eye view image features and second bird's-eye view image features, the initial student model is trained with feature difference training.

[0013] In an embodiment of the present application, through at least one of the training methods of first distillation training, second distillation training and feature difference training, the trained student model can learn the feature distribution of BEV from the teacher model, learn the feature distribution of the task head output of the initial teacher model, and distill the difference distribution of the student model with the difference distribution of the teacher model, so as to enable the student model to obtain the difference of BEV feature distribution between different tasks.

[0014] In some embodiments, based on multiple first bird's-eye image features and second bird's-eye image features, a first distillation training is performed on the initial student model, including: performing mean partitioning processing on the second bird's-eye image features of the initial student model to obtain second bird's-eye image sub-features whose number is the same as the number of first bird's-eye image features; constructing a first feature sample set including multiple first feature samples; each first feature sample includes a second bird's-eye image sub-feature, a first bird's-eye image feature as a first label for the second bird's-eye image sub-feature, and a preset second label; the second bird's-eye image sub-features and the first bird's-eye image features in any two first feature samples in the first feature sample set are different; based on the first feature sample set, the initial student model is subjected to a first distillation training.

[0015] In the embodiment of the present application, the initial student model is first distilled through the second bird's-eye view image sub-feature, the first bird's-eye view image feature as the first label of the second bird's-eye view image sub-feature, and the preset second label. In this way, each second bird's-eye view image sub-feature is distilled from the first bird's-eye view image feature of a teacher model, so that the feature distribution of the student model is consistent with that of the teacher model.

[0016] In some embodiments, the second feature map includes multiple second sub-feature maps corresponding one-to-one to multiple image reasoning tasks; based on the multiple first feature maps and the second feature maps, the initial student model is subjected to second distillation training, including: for each image reasoning task, the first feature map corresponding to the image reasoning task is used as the third label of the second sub-feature map corresponding to the image reasoning task, and the second sub-feature map, the third label, and the preset fourth label corresponding to the image reasoning task are used as the second feature samples of the image reasoning task; based on the second feature samples corresponding to the multiple image reasoning tasks, the initial student model is subjected to second distillation training.

[0017] In the embodiment of the present application, the initial student model is subjected to the second distillation training through the second sub-feature map, the first feature map as the label of the second sub-feature map, and the preset fourth label. In this way, each task head of the initial student model can accept the distillation learning of the task head of the initial teacher model corresponding to the task type of the task head, so that multiple task heads of the initial student model learn the feature distribution output by the task head of the initial teacher model, and then can perform the corresponding image reasoning task.

[0018] In some embodiments, feature difference training is performed on the initial student model based on multiple first bird's-eye image features and second bird's-eye image features, including: performing mean partitioning processing on the second bird's-eye image features of the initial student model to obtain second bird's-eye image sub-features whose number is the same as the number of first bird's-eye image features; determining the similarity between every two first bird's-eye image features in the multiple first bird's-eye image features to obtain a first similarity matrix; determining the similarity between every two second bird's-eye image sub-features in the multiple second bird's-eye image sub-features to obtain a second similarity matrix; based on the first similarity matrix and the second similarity matrix, feature difference training is performed on the initial student model.

[0019] In the embodiment of the present application, the similarity between every two first bird's-eye view image features is first determined to obtain a first similarity matrix, and then the similarity between every two second bird's-eye view image sub-features is determined to obtain a second similarity matrix. Finally, the initial student model is trained for feature difference using the first similarity matrix and the second similarity matrix. In this way, the distribution of each bird's-eye view image feature of the student model is made more similar to the distribution of the bird's-eye view image features of the teacher model, thereby improving the student model's ability to model bird's-eye view image features for multiple tasks.

[0020] In some embodiments, based on the first similarity matrix and the second similarity matrix, feature difference training is performed on the initial student model, including: determining the deviation information of the second similarity matrix relative to the first similarity matrix; based on the deviation information of the second similarity matrix relative to the first similarity matrix, feature difference training is performed on the initial student model.

[0021] In an embodiment of the present application, feature difference training is performed on the initial student model based on the deviation information of the second similarity matrix relative to the first similarity matrix. This can make the distribution of each bird's-eye view image feature of the student model more similar to the distribution of the bird's-eye view image feature of the teacher model, thereby improving the student model's ability to model bird's-eye view image features for multiple tasks.

[0022] In a second aspect, an embodiment of the present application provides an image processing method, the image processing method comprising:

[0023] Get the image to be processed;

[0024] The image to be processed is processed using an image processing model to obtain an image processing result; wherein the image processing model is obtained according to the training method of the multi-task model in the above embodiment.

[0025] In a third aspect, an embodiment of the present application provides a multi-task model training device, the multi-task model training device comprising:

[0026] A first acquisition module is used to acquire multiple initial teacher models corresponding to multiple image reasoning tasks, and an initial student model for performing multiple image reasoning tasks; the image reasoning tasks are reasoning tasks based on multiple environmental images of the vehicle; the number of channels of the images processed by each initial teacher model is the same, and the sum of the number of channels of the images processed by the multiple initial teacher models is the number of channels of the images processed by the initial student model;

[0027] A feature extraction module, used for determining a first image feature of each initial teacher model and a second image feature of an initial student model based on a preset set of environmental image samples;

[0028] The training module is used to train the initial student model based on the first image features corresponding to multiple initial teacher models and the second image features of the initial student model to obtain a trained student model.

[0029] In a fourth aspect, an embodiment of the present application provides an image processing device, the image processing device comprising:

[0030] A second acquisition module is used to acquire the image to be processed;

[0031] An image processing module is used to process the image to be processed using an image processing model to obtain an image processing result;

[0032] Among them, the image processing model is obtained according to the training method of the multi-task model in the above embodiment.

[0033] In a fifth aspect, an embodiment of the present application provides a training device for a multi-task model, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0034] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0035] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which implement some or all of the steps in the above method when the computer program or instructions are executed by a processor.

[0036] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0038] Figure 1a Schematic diagram 1 of generating a bird's-eye view in the related art;

[0039] Figure 1b Schematic diagram 2 of generating a bird's-eye view in the related art;

[0040] Figure 1c A schematic diagram of a multi-task model training method according to an embodiment of the present application;

[0041] Figure 2a A schematic diagram of the execution of the initial teacher model and the initial student model provided in the embodiment of the present application;

[0042] Figure 2b Schematic diagram 2 of an implementation flow of a multi-task model training method provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the implementation process of a multi-task model training method provided in an embodiment of the present application Figure 3 ;

[0044] Figure 4 A schematic diagram of the implementation process of a multi-task model training method provided in an embodiment of the present application Figure 4 ;

[0045] Figure 5 A schematic diagram of the implementation process of a multi-task model training method provided in an embodiment of the present application Figure 5 ;

[0046] Figure 6 A schematic diagram of the structure of a multi-task model training device provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of a hardware entity of a multi-task model training device in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0050] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0051] The terms "first / second / third" involved are merely used to distinguish similar vehicles and do not represent a specific order for the vehicles. It is understandable that "first / second / third" can be interchanged with the specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0053] In order to improve the efficiency and accuracy of the test driving system, an embodiment of the present application provides a training method for a multi-task model. The training method for the multi-task model can be applied to a test host computer of a training system for the multi-task model. The test host computer includes but is not limited to fixed devices and / or mobile devices. For example, the fixed devices include but are not limited to: a personal computer (PC), or a server, etc., and the server can be a cloud server or an ordinary server.

[0054] With the rapid development of autonomous driving technology, Bird's Eye View (BEV) has become one of the key technologies in autonomous driving systems. BEV perception generates a unified bird's eye view image by integrating data from multiple sensors, such as Figure 1a As shown, the BEV perception module 13 processes the image data 10 collected by the multi-view camera, the radar / lidar point cloud data 11 and other related sensor data 12 to obtain a bird's-eye view image 14. Therefore, BEV perception enables the autonomous driving system to understand the environment around the vehicle more intuitively and comprehensively, thereby greatly improving perception accuracy and decision-making capabilities.

[0055] BEV perception technology relies on multi-sensor data, including cameras, laser radar (LiDAR), radar, and ultrasonic sensors. Cameras provide high-resolution visual data, enabling the system to identify lane lines, traffic signs, and other visual features. LiDAR provides accurate depth information, especially in three-dimensional space, which can help the system perceive the shape and distance of objects. Radar performs well in various adverse weather conditions and provides reliable speed and distance data, especially in low-visibility environments such as rain and fog. Ultrasonic sensors are used to detect close objects and are usually used in parking and low-speed operation scenarios. The data from these sensors needs to be effectively fused to generate a comprehensive bird's-eye view. In order to achieve effective fusion of multi-sensor data, coordinate transformation is one of the key steps. The data collected by each sensor is usually in its own coordinate system, so these data must be projected onto a common reference plane, such as the ground plane. Through geometric transformation and data alignment technology, the data from each sensor can be fused to eliminate their respective deviations and generate a unified bird's-eye view. This process not only needs to consider the spatial position and posture of each sensor, but also needs to deal with the time synchronization problem between sensors to ensure the timeliness and accuracy of the fused data.

[0056] In BEV perception, it is necessary to extract data features from multiple sensors through advanced algorithms. Deep learning models, especially convolutional neural networks (CNN), play a vital role in this process. These models can efficiently identify and classify various objects in the environment, including vehicles, pedestrians, road signs, etc. By training on large amounts of data, deep learning models can extract key features in the environment, perform real-time identification and classification, and provide high-precision perception results.

[0057] In the BEV perception algorithm, visual BEV plays a particularly important role. The image features obtained by perceiving the surrounding environment through visual means are rich in layers and granularity. In addition, because the cost of cameras is relatively low, it is easy to meet the large-scale assembly of various models, and is regarded as the first choice for environmental perception tasks. Visual BEV generates bird's-eye images through high-resolution visual data provided by the camera. Compared with other sensors, visual BEV can provide rich color and texture information, which is very beneficial for identifying and classifying various objects in the environment. The camera can capture the appearance characteristics of objects, such as color, shape, texture, etc., so that the system can more accurately identify lane lines, traffic signs, pedestrians and other road users. In addition, the spatial resolution of visual BEV is high, which can present detailed information of the surrounding environment in detail. High-resolution image data enables the system to detect small and long-distance objects, thereby improving the accuracy and range of perception. In addition, the camera has an advantage in capturing dynamic objects, and can record the movement trajectory of vehicles and pedestrians in real time, helping the system to track and predict dynamic objects.

[0058] Visual BEV perception algorithms can be divided into three categories according to the model structure: Inverse Perspective Mapping (IPM)-based architecture method, pseudo point cloud prediction architecture method, and Transformer attention-based method.

[0059] Early BEV perception mainly relied on inverse perspective mapping (IPM) to transform the camera image according to the geometric relationship and project it to the BEV perspective. This method is the simplest and most efficient. However, its disadvantages are also prominent, that is, it assumes that the ground environment where the camera is located is flat. Only under this premise can the image after projection avoid distortion and easily produce deviations. In many current studies, in order to realize the conversion of 2D image information to the BEV perspective, a method based on pseudo point cloud projection has become the mainstream. The LSS (Lift / Splat / Shoot) method proposed in the paper "Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d" is the first pseudo point cloud projection. This method generates 3D features (Lift) from two-dimensional image features, then "flattens" the 3D features to obtain a BEV feature map (Splat), and finally performs related task operations (Shooting) on ​​the BEV feature map. Only an arbitrary number of camera inputs are required to obtain a good bird's-eye view result. The method is as follows Figure 1bAs shown, the method includes: S1, the backbone network model 20 performs depth prediction on the image collected by the 6-view camera layout to obtain a feature map; S2, generates a 6-camera view frustum point cloud according to the height and width of the output feature map; S3, the view frustum point cloud is converted to the vehicle coordinate system through internal and external references; S4, the pseudo point cloud is filtered and divided into grids; S5, voxel pooling operation; wherein, 1×c×BEV feature height×BEV feature width voxel pooling operation, one point cloud corresponds to a c-channel feature, all features falling into the same grid are fused to form a BEV feature; S6, BEV feature decoding network model 30, obtains the prediction result under the BEV perspective.

[0060] Among them, the key to the above steps is: the first step Lift: extract the features within the range of each camera's view cone; the second step Spalt: project all the features to the top view. The role of Lift is to extract the features of the pixels in the 2D image in the 3D world with the body coordinates as the origin. Generally speaking, if the 2D pixel coordinates are to be projected in the 3D world, the depth information of the pixel points is indispensable. Each pixel point in the image corresponds to a ray in the world coordinates. LSS generates a bunch of discrete depth values ​​for each pixel point. When the model is trained, the network itself selects the appropriate depth. Combined with the depth information, the first step is to generate a view cone point cloud. The role of this view cone point cloud is to explicitly guide the 2D image features to generate 3D BEV features according to this point cloud in the Splat stage. After Lift, the features of a 2D pixel point in the 3D space are obtained, and the 2D pixel coordinates and depth values ​​of the pixel are obtained. Combined with the intrinsic and extrinsic parameters of the camera, the 3D coordinates corresponding to the pixel in the body coordinate system can be calculated. The spatial scales of the view cone point cloud and the image features are equal, that is, the positions have a one-to-one correspondence. After the view cone point cloud is converted to the BEV perspective, each point will be assigned to the BEV grid to form the features under the BEV perspective. Based on this, the BEV features are subsequently decoded and connected to the task head for prediction of related tasks. The LSS-based method can construct BEV features through geometric relationships. The model has stronger modules and structural characteristics. After the BEV features are formed, it is convenient to connect various prediction tasks. The research of this patent is also related work based on the LSS architecture system. The information processing architecture based on the Transformer attention mechanism is also having a very important impact on the perception processing of intelligent driving BEV. This is mainly due to the fact that through a large number of sample learning, the Transformer model can intuitively establish the position association between the BEV view and the corresponding points in the vehicle-side camera image, especially its cross-attention can more conveniently describe a relationship between the corresponding feature content under different perspectives, thereby helping to mine the accurate semantic information corresponding to each corresponding position of the BEV view and the vehicle-side camera image. The learning method of transformer is difficult to achieve with traditional inverse projection transformation and convolutional neural network. Therefore, Transformer is regarded as an effective means to significantly improve the accuracy of BEV perception prediction to a certain extent, making it gradually become an inevitable and important mainstream trend in current research in this field.

[0061] Visual BEV plays an important role in multiple core application scenarios in autonomous driving. In terms of road and lane detection, the bird's-eye view can more accurately identify lane lines and road boundaries, significantly improving the vehicle's path tracking capability and driving safety. In terms of object detection and tracking, the use of visual BEV can efficiently detect and track dynamic objects such as vehicles and pedestrians to ensure driving safety and avoid collision accidents. In complex environments such as parking lots, visual BEV technology can accurately identify surrounding obstacles and vacant parking spaces, assisting vehicles in automatic parking operations. In terms of path planning and decision-making, by comprehensively utilizing a variety of information in the bird's-eye view, the autonomous driving system can perform more accurate and efficient path planning and real-time decision-making to ensure safety and efficiency during driving.

[0062] In autonomous driving systems, multi-task training is a common strategy, but in BEV multi-task training, there may be conflicts between different tasks. For example, the object detection task requires high-precision bounding box prediction, while the semantic segmentation task requires fine pixel-level classification.

[0063] Since the BEV representation is an intermediate representation shared by multiple tasks, different tasks may have different or even contradictory requirements for BEV representation. These conflicts not only affect the performance of the model, but also impose strict requirements on task priority and resource allocation. If there is no additional information guidance during the training process, it is easy to cause the model performance to degrade. In order to solve the above problems, the multi-task optimization of the model can be carried out through multi-model distillation learning.

[0064] In the related art, there is no method that can optimize visual BEV multi-tasks through distillation learning.

[0065] In order to solve the above technical problems, the present application embodiment provides a training method for a multi-task model, such as Figure 1c As shown, the method includes the following steps S101 to S103:

[0066] Step S101, obtain multiple initial teacher models corresponding to multiple image reasoning tasks, and an initial student model for executing the multiple image reasoning tasks; the image reasoning tasks are reasoning tasks based on multiple environmental images of the vehicle; the number of channels of the images processed by each of the initial teacher models is the same, and the sum of the number of channels of the images processed by the multiple initial teacher models is the number of channels of the images processed by the initial student model.

[0067] Here, the multiple environment images of the vehicle may be multi-perspective images of the vehicle collected by a multi-perspective camera of the vehicle, for example, six-perspective images.

[0068] In the embodiment of the present application, different initial teacher models can be constructed according to the types of different image reasoning tasks. For example, image reasoning tasks may include road and lane detection tasks, object detection and tracking tasks, and path planning and decision-making tasks. In other words, multiple environmental images of the vehicle are input into different initial teacher models to perform corresponding image reasoning tasks.

[0069] In the embodiment of the present application, the model structures of different initial teacher models correspond to their own image reasoning tasks. That is to say, when the image reasoning tasks are different, the model structures of the corresponding initial teacher models are also different. Among them, the different model structures of the initial teacher models refer to the different backbone network and decoding network modules.

[0070] In some embodiments, the model capacity of the initial teacher model is greater than the model capacity of the initial student model.

[0071] In some embodiments, the initial teacher model further includes a task head, the model structures of the task heads of the multiple initial teacher models are the same, and the number of channels of the image processed by each initial teacher model is the same. Exemplarily, the number of channels of the image processed by the initial teacher model can be 3, namely, RGB channels.

[0072] In the embodiment of the present application, the sum of the number of channels of the images processed by the multiple initial teacher models is the number of channels of the image processed by the initial student model. For example, when the number of initial teacher models is 3 and the number of channels is 3, the number of channels of the image processed by the initial student model is 9.

[0073] Step S102, based on a preset set of environmental image samples, determining the first image features of each of the initial teacher models and the second image features of the initial student models.

[0074] In an embodiment of the present application, the environmental image sample set can be input into each initial teacher model to obtain the first image features corresponding to multiple initial teacher models respectively, and the environmental image sample set can be input into the initial student model to obtain the second image features of the initial student model.

[0075] In some embodiments, each initial teacher model and initial student model includes a backbone network and a task head; the environmental image sample set is input into each initial teacher model, and the backbone network is used to extract features of the environmental image sample set to obtain the first bird's-eye view image features of the initial teacher model, and then the first bird's-eye view image features are processed by the task head to obtain the first feature map of the initial teacher model; similarly for the initial student model, the environmental image sample set is input into the initial student model, and the backbone network is used to extract features of the environmental image sample set to obtain the second bird's-eye view image features of the initial student model, and then the second bird's-eye view image features are processed by the task head to obtain the second feature map of the initial student model.

[0076] Step S103, training the initial student model based on the first image features corresponding to the multiple initial teacher models and the second image features of the initial student model to obtain a trained student model.

[0077] In an embodiment of the present application, the first image feature of the initial teacher model can be used as the soft label information of the second image feature, and then the initial student model can be distilled and trained. In some embodiments, the preset real label information can also be obtained, and the model parameters of the initial student model are updated by comparing the prediction error of the second image feature to the real label information and the matching degree of the second image feature to the soft label information, until the prediction error and the matching degree meet the preset conditions, or the number of iterations meets the preset number, that is, the distillation training of the initial student model is completed.

[0078] In an embodiment of the present application, an initial teacher model and an initial student model corresponding to different image reasoning tasks are first constructed; wherein the number of channels of the image processed by each initial teacher model is the same, and the sum of the number of channels of the images processed by multiple initial teacher models is the number of channels of the image processed by the initial student model. Then the first image feature of each initial teacher model and the second image feature of the initial student model are determined. Finally, based on the first image features corresponding to the multiple initial teacher models and the second image features of the initial student model, the initial student model is trained to obtain a trained student model. In this way, by training the initial student model with the first image features output by different initial teacher models, the initial student model can learn the feature distributions of different initial teacher models, thereby enabling a single student model to learn the differentiation between different tasks from multiple teacher models, and further enabling the trained student model to perform multiple image reasoning tasks.

[0079] In some embodiments, the above step S103 may also be implemented by at least one of steps S201 to S203:

[0080] Step S201: performing a first distillation training on the initial student model based on a plurality of the first bird's-eye view image features and the second bird's-eye view image features.

[0081] In the embodiment of the present application, both the initial teacher model and the initial student model include a backbone network. Therefore, by extracting features from the environmental image sample set through the backbone network of the initial teacher model, the first bird's-eye view image feature can be obtained; by extracting features from the environmental image sample set through the backbone network of the initial student model, the second bird's-eye view image feature can be obtained. At this time, the number of channels of each of the multiple first bird's-eye view image features is the same, and the sum of the number of channels of the multiple first bird's-eye view image features is equal to the number of channels of the second bird's-eye view image feature.

[0082] In some embodiments, multiple first bird's-eye view image features may be stacked to obtain a stacked bird's-eye view image feature, and then the deviation between the second bird's-eye view image feature and the stacked bird's-eye view image feature is determined, so as to adjust the model parameters of the backbone network in the initial student model according to the deviation. In this way, the initial student model can learn the distribution of the bird's-eye view image features of the initial teacher model.

[0083] In some embodiments, a preset second label can also be obtained, where the preset second label belongs to the real label information, and the deviation between the second bird's-eye view image feature and the second label is determined, so that the model parameters of the backbone network in the initial student model are adjusted according to the deviation.

[0084] Step S202: performing a second distillation training on the initial student model based on the plurality of the first feature maps and the second feature maps.

[0085] In the embodiment of the present application, both the initial teacher model and the initial student model include task heads. Because one initial teacher model is responsible for one image reasoning task, and the initial student model is responsible for multiple image reasoning tasks. Therefore, the sum of the number of task heads of multiple initial teacher models is equal to the number of task heads of the initial student model. In this way, the model parameters of each task head of the initial student model can be adjusted through multiple first feature maps.

[0086] In the embodiment of the present application, the second feature map may include the second sub-feature map output by each task head in the initial student model. Then, multiple first feature maps and multiple second sub-feature maps are combined in pairs, that is, a first feature map is used as label information of a second sub-feature map, and the deviation between the first feature map and the second sub-feature map is determined, and the model parameters of each task head in the initial student model are adjusted by the deviation.

[0087] In some embodiments, a preset fourth label can also be obtained, where the preset fourth label belongs to the real label information. By determining the deviation between the second sub-feature map and the fourth label, the model parameters of each task head in the initial student model are adjusted according to the deviation.

[0088] For example, Figure 2a As shown, the initial teacher model and the initial student model include a backbone network 201, a decoding module 203 and a task head 204. It should be noted that in actual application, because an initial teacher model is only responsible for one image reasoning task, an initial teacher model only includes one task head, and the initial student model is responsible for multiple image reasoning tasks, so multiple initial student models include multiple task heads. The model input 205 of the backbone network 201 includes 6-view images collected by a 6-view camera layout, camera internal parameters, and camera external parameters relative to the vehicle or radar. The backbone network 201 can extract the image features of the 6-view images to obtain the BEV features 202 (i.e., the above-mentioned bird's-eye view image features); then the decoding module 203 can decode the BEV features 202, and then the task head 204 can output feature maps based on the decoded BEV features 202.

[0089] Step S203: performing feature difference training on the initial student model based on a plurality of the first bird's-eye view image features and the second bird's-eye view image features.

[0090] In the embodiment of the present application, the second bird's-eye view image feature is first divided according to the number of channels of the first bird's-eye view image feature to obtain the second bird's-eye view image sub-features having the same number as the first bird's-eye view image feature. Then, based on the difference information between the multiple first bird's-eye view image features and the difference information between the multiple second bird's-eye view image sub-features, the initial student model is trained for feature difference.

[0091] In an embodiment of the present application, through at least one of the training methods of first distillation training, second distillation training and feature difference training, the trained student model can learn the feature distribution of BEV from the teacher model, learn the feature distribution of the task head output of the initial teacher model, and distill the difference distribution of the student model with the difference distribution of the teacher model, so as to enable the student model to obtain the difference of BEV feature distribution between different tasks.

[0092] In some embodiments, Figure 2b As shown, the above step S201 can be implemented through steps S301 to S303:

[0093] Step S301 , performing mean division processing on the second bird's-eye view image features of the initial student model to obtain second bird's-eye view image sub-features whose number is the same as that of the first bird's-eye view image features.

[0094] In an embodiment of the present application, the second bird's-eye view image feature of the initial student model can be mean-partitioned according to the number of channels of the first bird's-eye view image feature of the initial teacher model to obtain multiple second bird's-eye view image sub-features.

[0095] Exemplarily, the number of initial teacher models is 3, the number of channels of the first bird's-eye view image feature of the initial teacher model is 3, and the number of channels of the second bird's-eye view image feature of the initial student model is 9. Then, the second bird's-eye view image feature can be divided into 3 second bird's-eye view image sub-features, and the number of channels of each second bird's-eye view image sub-feature is 3.

[0096] Step S302, constructing a first feature sample set including multiple first feature samples; each of the first feature samples includes the second bird's-eye view image sub-feature, the first bird's-eye view image feature as the first label of the second bird's-eye view image sub-feature, and a preset second label; the second bird's-eye view image sub-feature and the first bird's-eye view image feature in any two of the first feature samples in the first feature sample set are different.

[0097] In an embodiment of the present application, a first bird's-eye view image feature among multiple first bird's-eye view image features can be used as a first label (i.e., a soft label) of a second bird's-eye view image sub-feature among multiple second bird's-eye view image sub-features, and then the preset second label can also be used as the real label of the second bird's-eye view image sub-feature. In other words, each second bird's-eye view image sub-feature includes two labels, namely the second bird's-eye view image sub-feature and the preset second label, and finally a first feature sample set of multiple first feature samples is obtained. In this way, each second bird's-eye view image sub-feature undergoes distillation learning of the first bird's-eye view image feature of a teacher model, with the aim of causing the feature distribution of the student model to be consistent with that of the teacher model.

[0098] Exemplarily, the multiple first bird's-eye image features include a first bird's-eye image feature 1 and a first bird's-eye image feature 2, and the multiple second bird's-eye image sub-features include a second bird's-eye image sub-feature 1 and a second bird's-eye image sub-feature 2. The first bird's-eye image feature 1 can be used as the first label of the second bird's-eye image sub-feature 1, and the first bird's-eye image feature 2 can be used as the first label of the second bird's-eye image sub-feature 1, and the first bird's-eye image feature 1 can be used as the first label of the second bird's-eye image sub-feature 2. Alternatively, the first bird's-eye image feature 2 can be used as the first label of the second bird's-eye image sub-feature 1, and the first bird's-eye image feature 1 can be used as the first label of the second bird's-eye image sub-feature 2.

[0099] Step S303: performing a first distillation training on the initial student model based on the first feature sample set.

[0100] In the embodiment of the present application, the deviation between the second bird's-eye view image sub-feature and the first bird's-eye view image feature as the first label of the second bird's-eye view image sub-feature, and the deviation between the second bird's-eye view image sub-feature and the preset second label can be determined, and the model parameters of the backbone network of the initial student model can be adjusted by the deviation. The above deviation can be represented by L1 loss, L2 loss, and KL divergence.

[0101] In the embodiment of the present application, the initial student model is first distilled through the second bird's-eye view image sub-feature, the first bird's-eye view image feature as the first label of the second bird's-eye view image sub-feature, and the preset second label. In this way, each second bird's-eye view image sub-feature is distilled from the first bird's-eye view image feature of a teacher model, so that the feature distribution of the student model is consistent with that of the teacher model.

[0102] In some embodiments, the second feature map includes a plurality of second sub-feature maps corresponding to a plurality of image reasoning tasks one by one; the above step S202 may be implemented by steps S2021 and S2022:

[0103] Step S2021, for each of the image reasoning tasks, the first feature map corresponding to the image reasoning task is used as the third label of the second sub-feature map corresponding to the image reasoning task, and the second sub-feature map corresponding to the image reasoning task, the third label, and the preset fourth label are used as the second feature sample of the image reasoning task.

[0104] In an embodiment of the present application, for different image reasoning tasks, the task head of the initial teacher model corresponding to the image reasoning task can output a first feature map corresponding to the image reasoning task, and the task head of the initial teacher model corresponding to the image reasoning task can output a second sub-feature map corresponding to the image reasoning task. Therefore, in order to enable the task head of the initial student model to learn the feature distribution output by the task head of the initial teacher model, the first feature map of the initial teacher model can be used as a label for the second sub-feature map of the initial student model.

[0105] Step S2022: performing a second distillation training on the initial student model based on second feature samples corresponding to the multiple image reasoning tasks respectively.

[0106] In an embodiment of the present application, for the task head corresponding to the image reasoning task in the initial student model, the first feature map output by the task head of the initial teacher model corresponding to the image reasoning task, and the second sub-feature map output by the task head corresponding to the image reasoning task in the initial student model and the preset fourth label can be used to adjust the model parameters of the task head corresponding to the image reasoning task in the initial student model. Among them, the deviation between the second sub-feature map and the first feature map of the third label of the second sub-feature map, and the deviation between the second sub-feature map and the preset fourth label can be determined, and the model parameters of the task head corresponding to the image reasoning task in the initial student model are adjusted by the deviation. Among them, the above deviation can be represented by L1 loss, L2 loss, and KL divergence.

[0107] In the embodiment of the present application, the initial student model is subjected to the second distillation training through the second sub-feature map, the first feature map as the label of the second sub-feature map, and the preset fourth label. In this way, each task head of the initial student model can accept the distillation learning of the task head of the initial teacher model corresponding to the task type of the task head, so that multiple task heads of the initial student model learn the feature distribution output by the task head of the initial teacher model, and then can perform the corresponding image reasoning task.

[0108] In some embodiments, Figure 3 As shown, the above step S203 can be implemented through steps S401 to S404:

[0109] Step S401 , performing mean division processing on the second bird's-eye view image features of the initial student model to obtain second bird's-eye view image sub-features whose number is the same as that of the first bird's-eye view image features.

[0110] In the embodiment of the present application, the implementation process of step S401 can refer to the above-mentioned step S301.

[0111] Step S402: determine the similarity between every two first bird's-eye view image features among the plurality of first bird's-eye view image features to obtain a first similarity matrix.

[0112] In some embodiments, the plurality of first bird's-eye view image features may be normalized so that the plurality of first bird's-eye view image features are between [0, 1]. For example, the plurality of first bird's-eye view image features may be normalized by performing a sigmoid function process.

[0113] In some embodiments, the similarity between every two first bird's-eye view image features after normalization may be determined, and then the similarity between every two first bird's-eye view image features may be determined as an element of a first similarity matrix, thereby constructing the first similarity matrix.

[0114] Step S403: determine the similarity between every two second bird's-eye view image sub-features in the plurality of second bird's-eye view image sub-features to obtain a second similarity matrix.

[0115] In some embodiments, the plurality of second bird's-eye view image sub-features may be normalized first, so that the plurality of second bird's-eye view image sub-features are between [0, 1]. For example, the plurality of second bird's-eye view image sub-features may be normalized by performing a sigmoid function process.

[0116] In some embodiments, the similarity between every two second bird's-eye view image sub-features after normalization may be determined, and then the similarity between every two second bird's-eye view image sub-features may be determined as an element of a second similarity matrix, thereby constructing the second similarity matrix.

[0117] Step S404: performing feature difference training on the initial student model based on the first similarity matrix and the second similarity matrix.

[0118] In an embodiment of the present application, the first similarity matrix can be used as label information of the second similarity matrix to perform feature difference training on the initial student model.

[0119] In the embodiment of the present application, the similarity between every two first bird's-eye view image features is first determined to obtain a first similarity matrix, and then the similarity between every two second bird's-eye view image sub-features is determined to obtain a second similarity matrix. Finally, the initial student model is trained for feature difference using the first similarity matrix and the second similarity matrix. In this way, the distribution of each bird's-eye view image feature of the student model is made more similar to the distribution of the bird's-eye view image features of the teacher model, thereby improving the student model's ability to model bird's-eye view image features for multiple tasks.

[0120] In some embodiments, the above step S404 may be implemented by steps S501 and S502:

[0121] Step S501: Determine deviation information of the second similarity matrix relative to the first similarity matrix.

[0122] In an embodiment of the present application, for each element of the second similarity matrix, the deviation between the elements of the second similarity matrix and the elements of the first similarity matrix is ​​determined, and the positions of the elements of the second similarity matrix in the second similarity matrix are the same as the positions of the elements of the first similarity matrix in the first similarity matrix, and then the deviation information between the elements corresponding to the positions of each element of the second similarity matrix relative to the first similarity matrix is ​​obtained.

[0123] Step S502: Based on the deviation information of the second similarity matrix relative to the first similarity matrix, feature difference training is performed on the initial student model.

[0124] In an embodiment of the present application, when the deviation information of the second similarity matrix relative to the first similarity matrix does not meet the preset deviation condition, the model parameters of the initial student model can be adjusted, and then iteration can be continued until the deviation information of the second similarity matrix relative to the first similarity matrix meets the preset deviation condition.

[0125] In an embodiment of the present application, feature difference training is performed on the initial student model through the deviation information of the second similarity matrix relative to the first similarity matrix, so that the distribution of each bird's-eye view image feature of the student model can be more similar to the distribution of the bird's-eye view image feature of the teacher model, thereby improving the student model's ability to model bird's-eye view image features for multiple tasks.

[0126] In some embodiments, Figure 4 As shown, the embodiment of the present application provides an image processing method, which can be implemented by steps S601 and S602:

[0127] Step S601, obtaining an image to be processed;

[0128] Step S602: Process the image to be processed using the image processing model to obtain an image processing result.

[0129] Among them, the above-mentioned image processing model is obtained based on the training method of the multi-task model provided in the above-mentioned embodiment.

[0130] In some embodiments, the present application provides a method for training a multi-task model. Before describing the method for training a multi-task model, the relevant variables are first defined:

[0131] B, N, C, H, and W represent the batch size of the model input image, the number of cameras, the image channels, the image height in pixels, and the image width in pixels, respectively. Batch size refers to the number of samples selected for one training.

[0132] H bev and W bev The table shows the height and width of the BEV feature. Among them, the BEV feature includes six image features extracted from different viewing angles by a deep network model. It belongs to the prior art;

[0133] S represents the student model, T n Represents the nth task teacher model, where 1 <n≤N。

[0134] F represents the feature, represents the BEV features of the student model, represents the BEV features of the nth teacher model, Represents the feature map output by the nth task head of the student model, Represents the feature map output by the task head of the nth teacher model, The dimensions and The dimensions are consistent.

[0135] represents the number of BEV channels of the student model, Indicates the number of BEV channels of the teacher model. The number of BEV channels of each teacher model is the same.

[0136] In the embodiment of the present application, the training method of the above multi-task model can be implemented by the following steps:

[0137] Step 1. First, you need to complete the training of the teacher model for N tasks (the initial teacher model mentioned above).

[0138] The encoding and decoding model structures of the teacher models for different tasks are constructed according to the tasks they are responsible for. There is no need to force consistency between the tasks. The teacher model should choose a model with a larger capacity as much as possible to ensure that the teacher model has a better prediction effect for the tasks it is responsible for.

[0139] Step 2. Build the student model (the initial student model mentioned above).

[0140] Among them, it is necessary to make the number of channels of the BEV feature of the student model (the above-mentioned bird's-eye view image feature) consistent with the sum of the number of channels of the BEV features of each teacher model, that is, At the same time, the dimension of the feature map output by the nth task head of the student model is made consistent with the dimension of the feature map output by the nth task head of the teacher model.

[0141] Step 3. After completing the training of the teacher model and building the student model, you can start the training of distillation learning.

[0142] Among them, distillation learning is started from three aspects: distillation of BEV features, distillation of output task heads, and distillation of BEV feature differences. Step 3 can be achieved by following the steps below:

[0143] Step 31, distillation of BEV features.

[0144] like Figure 5 As shown, the input data is input into the student backbone network 501, the task 1 teacher backbone network 1, the task 2 teacher backbone network 2... the task n teacher backbone network n, and a plurality of BEV features are obtained respectively;

[0145] First, the BEV features output by the student backbone network 501 Perform N-means division from the channel dimension to obtain N BEV features, which correspond one-to-one to the BEV features of the teacher model.

[0146] Each BEV feature is distilled from the BEV feature of a teacher model, with the goal of making the feature distribution of the student model consistent with that of the teacher model. The loss functions used for distillation include but are not limited to L1 loss, L2 loss, KL divergence, etc., depending on the experimental results.

[0147] Step 32, output the distillation of the task header.

[0148] like Figure 5 As shown, the N BEV features after mean division are output to the task head 41, task head 42...task head 41 of the student model through the student decoding module 502, and then the task head 41, task head 42...task head 41 output the feature map; the BEV features output by the backbone network 1 of the task 1 teacher are output to the task head 31 through the decoding module 21 of the task 1 teacher, and the task head 31 outputs the feature map; the BEV features output by the backbone network 2 of the task 2 teacher are output to the task head 32 through the decoding module 22 of the task 2 teacher, and the task head 32 outputs the feature map... The BEV features output by the backbone network n of the task N teacher are output to the task head 3n through the decoding module 2n of the task N teacher, and the task head 3n outputs the feature map. The feature map output by the task head is distilled before the decoding operation. The task head of the student model corresponds to the teacher model one by one, and the feature map output by the task head of the teacher model is used to distill the student model. The loss function used for distillation includes but is not limited to L1 loss, L2 loss, KL divergence, etc., depending on the experimental results.

[0149] Step 33, BEV feature difference distillation.

[0150] During the training process, a BEV feature difference distillation learning module is constructed.

[0151] First, the BEV feature difference distillation learning module counts the similarities between the BEV features of each teacher model. After being processed by the sigmoid function, it is scaled to [0,1] and obtained Then the similarity of the BEV features of the i-th and j-th teacher models is counted and recorded as Then the similarity matrix shown in formula (1) can be formed:

[0152]

[0153] Similarly, the similarity matrix of the BEV features of the student model after being divided into N parts can be obtained, as shown in formula (2):

[0154]

[0155] After having the similarity matrix of the two, P T As a guide, arrange the student model so that P S The value is closer to P T , so that the distribution of N BEV features of the student model is more similar to that of the teacher model, thereby improving the student model's ability to model BEV features for multiple tasks.

[0156] In some embodiments, Figure 6 As shown, the embodiment of the present application provides a multi-task model training device 600, which includes a first acquisition module 601, a feature extraction module 602 and a training module 603; wherein:

[0157] The first acquisition module 601 is used to acquire a plurality of initial teacher models corresponding to a plurality of image reasoning tasks, and an initial student model for executing the plurality of image reasoning tasks; the image reasoning tasks are reasoning tasks based on a plurality of environmental images of the vehicle; the number of channels of the images processed by each of the initial teacher models is the same, and the sum of the number of channels of the images processed by the plurality of initial teacher models is the number of channels of the images processed by the initial student model;

[0158] A feature extraction module 602, used to determine a first image feature of each of the initial teacher models and a second image feature of the initial student model based on a preset set of environmental image samples;

[0159] The training module 603 is used to train the initial student model based on the first image features corresponding to multiple initial teacher models and the second image features of the initial student model to obtain a trained student model.

[0160] In some embodiments, each of the first image features includes a first bird's-eye view image feature of the initial teacher model and a first feature map of the initial teacher model; the second image feature includes a second bird's-eye view image feature of the initial student model and a second feature map of the initial student model; the training module 603 is also used for at least one of the following: performing a first distillation training on the initial student model based on multiple first bird's-eye view image features and the second bird's-eye view image features; performing a second distillation training on the initial student model based on multiple first feature maps and the second feature maps; performing feature difference training on the initial student model based on multiple first bird's-eye view image features and the second bird's-eye view image features.

[0161] In some embodiments, the training module 603 is also used to perform mean partitioning processing on the second bird's-eye image feature of the initial student model to obtain the same number of second bird's-eye image sub-features as the first bird's-eye image features; construct a first feature sample set including multiple first feature samples; each of the first feature samples includes the second bird's-eye image sub-feature, the first bird's-eye image feature as the first label of the second bird's-eye image sub-feature, and a preset second label; the second bird's-eye image sub-feature and the first bird's-eye image feature in any two of the first feature samples in the first feature sample set are different; based on the first feature sample set, the initial student model is subjected to first distillation training.

[0162] In some embodiments, the second feature map includes multiple second sub-feature maps corresponding to multiple image reasoning tasks one by one; the training module 603 is also used to, for each of the image reasoning tasks, use the first feature map corresponding to the image reasoning task as the third label of the second sub-feature map corresponding to the image reasoning task, and use the second sub-feature map corresponding to the image reasoning task, the third label, and the preset fourth label as the second feature sample of the image reasoning task; based on the second feature samples corresponding to the multiple image reasoning tasks respectively, perform a second distillation training on the initial student model.

[0163] In some embodiments, the training module 603 is also used to perform mean partitioning processing on the second bird's-eye view image features of the initial student model to obtain the same number of second bird's-eye view image sub-features as the first bird's-eye view image features; determine the similarity between every two first bird's-eye view image features in multiple first bird's-eye view image features to obtain a first similarity matrix; determine the similarity between every two second bird's-eye view image sub-features in multiple second bird's-eye view image sub-features to obtain a second similarity matrix; and perform feature difference training on the initial student model based on the first similarity matrix and the second similarity matrix.

[0164] In some embodiments, the training module 603 is further used to determine the deviation information of the second similarity matrix relative to the first similarity matrix; based on the deviation information of the second similarity matrix relative to the first similarity matrix, perform feature difference training on the initial student model.

[0165] In some embodiments, Figure 7 As shown, the embodiment of the present application provides an image processing device 700, which includes a second acquisition module 701 and an image processing module 702; wherein:

[0166] The second acquisition module 701 is used to acquire the image to be processed;

[0167] The image processing module 702 is used to process the image to be processed using an image processing model to obtain an image processing result; wherein the above-mentioned image processing model is obtained according to the training method of the multi-task model in the above-mentioned embodiment.

[0168] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0169] It should be noted that in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0170] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0171] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0172] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0173] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0174] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0175] The present application embodiment provides a multi-task model training device, Figure 8 A schematic diagram of the structure of a multi-task model training device 800 provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the device includes: a processor 801, a communication interface 802 and a memory 803, wherein:

[0176] The processor 801 generally controls the overall operation of the multi-task model training device 800, and the overall operation may be to implement the multi-task model training method provided in the embodiment of the present application.

[0177] The communication interface 802 enables the multi-task model training device 800 to communicate with other terminals or servers through the network.

[0178] The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or processed by each module in the processor 801 and the training device 800 of the multi-task model (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 801, the communication interface 802, and the memory 803 through the bus 804.

[0179] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a readable storage medium. The processor of the computer device reads the computer instructions from the readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method of the multi-task model described in the embodiment of the present application.

[0180] An embodiment of the present application provides a readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the training method of the multi-task model provided in the embodiment of the present application.

[0181] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0182] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0183] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0184] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0185] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0187] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0188] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0189] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0190] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0191] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A training method for a multi-task model, characterized in that: The training method comprises: Acquire multiple initial teacher models corresponding to multiple image reasoning tasks, and an initial student model for executing the multiple image reasoning tasks; the image reasoning tasks are reasoning tasks based on multiple environmental images of the vehicle; the number of channels of the images processed by each of the initial teacher models is the same, and the sum of the number of channels of the images processed by the multiple initial teacher models is the number of channels of the images processed by the initial student model; Based on a preset set of environmental image samples, determining a first image feature of each of the initial teacher models and a second image feature of the initial student model; Based on the first image features corresponding to multiple initial teacher models and the second image features of the initial student model, the initial student model is trained to obtain a trained student model.

2. The training method according to claim 1, characterized in that: Each of the first image features includes a first bird's-eye view image feature of the initial teacher model and a first feature map of the initial teacher model; each of the second image features includes a second bird's-eye view image feature of the initial student model and a second feature map of the initial student model; The training of the initial student model based on the first image features respectively corresponding to the multiple initial teacher models and the second image features of the initial student model comprises at least one of the following: Based on the first bird's-eye view image features and the second bird's-eye view image features, performing a first distillation training on the initial student model; Based on the first feature maps and the second feature maps, performing a second distillation training on the initial student model; Based on the first bird's-eye view image features and the second bird's-eye view image features, feature difference training is performed on the initial student model.

3. The training method according to claim 2, characterized in that: The performing a first distillation training on the initial student model based on the plurality of first bird's-eye view image features and the second bird's-eye view image features comprises: Performing mean division processing on the second bird's-eye view image features of the initial student model to obtain second bird's-eye view image sub-features whose number is the same as that of the first bird's-eye view image features; Constructing a first feature sample set including a plurality of first feature samples; each of the first feature samples includes the second bird's-eye view image sub-feature, the first bird's-eye view image feature as a first label of the second bird's-eye view image sub-feature, and a preset second label; the second bird's-eye view image sub-feature and the first bird's-eye view image feature in any two of the first feature samples in the first feature sample set are different; Based on the first feature sample set, a first distillation training is performed on the initial student model.

4. The training method according to claim 2, characterized in that: The second feature map includes a plurality of second sub-feature maps corresponding to a plurality of image reasoning tasks one by one; and performing a second distillation training on the initial student model based on the plurality of first feature maps and the second feature maps includes: For each of the image reasoning tasks, using the first feature map corresponding to the image reasoning task as the third label of the second sub-feature map corresponding to the image reasoning task, and using the second sub-feature map corresponding to the image reasoning task, the third label, and a preset fourth label as the second feature sample of the image reasoning task; Based on second feature samples respectively corresponding to the multiple image reasoning tasks, the initial student model is subjected to second distillation training.

5. The training method according to claim 2, characterized in that: The performing feature difference training on the initial student model based on the plurality of first bird's-eye view image features and the second bird's-eye view image features comprises: Performing mean division processing on the second bird's-eye view image features of the initial student model to obtain second bird's-eye view image sub-features whose number is the same as that of the first bird's-eye view image features; Determine the similarity between every two first bird's-eye view image features among the plurality of first bird's-eye view image features to obtain a first similarity matrix; determining a similarity between every two second bird's-eye view image sub-features in the plurality of second bird's-eye view image sub-features to obtain a second similarity matrix; Based on the first similarity matrix and the second similarity matrix, feature difference training is performed on the initial student model.

6. The training method according to claim 5, characterized in that: The performing feature difference training on the initial student model based on the first similarity matrix and the second similarity matrix includes: Determining deviation information of the second similarity matrix relative to the first similarity matrix; Based on the deviation information of the second similarity matrix relative to the first similarity matrix, feature difference training is performed on the initial student model.

7. An image processing method, characterized in that: include: Get the image to be processed; Processing the image to be processed by using the image processing model to obtain an image processing result; Wherein, the image processing model is obtained according to the training method of the multi-task model described in any one of claims 1 to 6.

8. A multi-task model training device, characterized in that: The training device comprises: A first acquisition module is used to acquire a plurality of initial teacher models corresponding to a plurality of image reasoning tasks, and an initial student model for executing the plurality of image reasoning tasks; the image reasoning tasks are reasoning tasks based on a plurality of environmental images of the vehicle; the number of channels of the images processed by each of the initial teacher models is the same, and the sum of the number of channels of the images processed by the plurality of initial teacher models is the number of channels of the images processed by the initial student model; A feature extraction module, used for determining a first image feature of each of the initial teacher models and a second image feature of the initial student model based on a preset set of environmental image samples; The training module is used to train the initial student model based on the first image features corresponding to multiple initial teacher models and the second image features of the initial student model to obtain a trained student model.

9. An image processing device, characterized in that: The image processing device comprises: A second acquisition module, used for acquiring an image to be processed; An image processing module is used to process the image to be processed using an image processing model to obtain an image processing result; Wherein, the image processing model is obtained according to the training method of the multi-task model described in any one of claims 1 to 6.

10. A multi-task model training device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the steps in the multi-task model training method described in any one of claims 1 to 7 when executing the executable instructions stored in the memory.

Images