Image processing model training method and device, equipment and storage medium

Abstract

The application discloses a kind of training method, device and equipment of image processing model and storage medium.The training method of image processing model includes: obtaining the sample image related to several kinds of perception tasks of vehicle and the sample label corresponding to each sample image;Using image processing model, each sample image is carried out image processing, and the first prediction result and the second prediction result of each sample image are obtained;Based on the first difference between the first prediction result of each sample image and the annotation result in each sample label, and the second difference between the second prediction result of each sample image and the sample identification in each sample label, determine target loss;Using target loss adjusts the parameter in image processing model, and the image processing model of training completion is obtained.The above scheme can improve the prediction performance of the image processing model of training completion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automobiles, and in particular to a training method and device of an image processing model, an equipment and a storage medium. BACKGROUND

[0002] The perception task in the field of automatic driving usually uses multiple sensors to perceive various environmental information around the vehicle body, and then processes the sensor data through the computer vision technology which is more popular in recent years to obtain understanding information of the environment around the vehicle body. Different network structures are usually set for different perception tasks to make corresponding result prediction. In order to improve the prediction performance of the model, a relatively complex network structure is designed, which leads to the failure to meet the real-time processing requirements of the system. Therefore, designing a same network model to perceive and predict multiple perception tasks is the development mainstream of the current automatic driving technology. However, for the training of the network model for simultaneously perceiving and predicting multiple perception tasks, the network layer of a single perception task is usually trained one by one, and then the trained network model is obtained through joint training of multiple tasks. In this case, the gradient contribution of different perception tasks to the network shared layer is inconsistent, which leads to poor prediction performance of the trained network model.

[0003] In view of the existing technical defects, how to provide a model training scheme with better prediction performance is a technical problem to be solved by the person skilled in the art. SUMMARY

[0004] The present application provides at least a training method, device and equipment of an image processing model, and a storage medium.

[0005] The present application provides a training method of an image processing model, comprising: obtaining sample images related to a plurality of perception tasks of a vehicle and sample labels corresponding to each sample image, wherein the sample label of each sample image comprises a sample identifier of a perception task to which the sample image belongs and a labeling result related to the perception task in the sample image; performing image processing on each sample image by using an image processing model to obtain first prediction results and second prediction results of each sample image, wherein the first prediction result of each sample image is used to represent the prediction result of the sample image on at least one perception task, and the second prediction result of each sample image is used to represent the perception task to which the sample image belongs; determining a target loss based on a first difference between the first prediction result of each sample image and the labeling result in each sample label, and a second difference between the second prediction result of each sample image and the sample identifier in each sample label; and adjusting the parameters in the image processing model by using the target loss to obtain a trained image processing model.

[0006] In some embodiments, the image processing model is used to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, including: performing feature extraction on each sample image to obtain sample features corresponding to each sample image; for each sample image, inputting the sample features corresponding to the sample image into an image processing network related to the perception task to which the sample image belongs to obtain a first prediction result corresponding to the sample image; and inputting the sample features corresponding to each sample image into a preset task classifier respectively to obtain a second prediction result corresponding to each sample image.

[0007] In some embodiments, the inputting of the sample features corresponding to each sample image into the preset task classifier to obtain the second prediction result corresponding to each sample image includes: for each sample image, performing the following steps: performing convolution processing on the sample features of the sample image to obtain convolution features of the sample image; performing dimension transformation processing on the convolution features of the sample image to obtain transformed features corresponding to the sample image; and determining the second prediction result corresponding to the sample image based on the transformed features corresponding to the sample image.

[0008] In some embodiments, the dimension transformation processing on the convolution features of the sample image to obtain the transformed features corresponding to the sample image includes: performing a global average pooling operation on the convolution features of the sample image to obtain pooled features corresponding to the sample image; performing first dimension transformation processing on the pooled features to obtain first dimension transformed features corresponding to the sample image; and performing second dimension transformation processing on the first dimension transformed features corresponding to the sample image to obtain second dimension transformed features corresponding to the sample image, and taking the second dimension transformed features corresponding to the sample image as the transformed features corresponding to the sample image.

[0009] In some embodiments, the step of performing convolution processing on the sample features of the sample image to obtain the convolution features of the sample image is performed by a preset convolution module in the image processing model, the preset convolution module includes a plurality of preset convolution layers arranged in cascade, the number of the plurality of preset convolution layers is the same as the number of the perception tasks, and the features output by the last preset convolution layer are taken as the convolution features of the sample image, the preset convolution module performs the convolution processing on the sample features of the sample image input into the preset convolution module to obtain the convolution features of the sample image in the following manner: taking the sample features of the sample image or the features output by the last preset convolution layer as the input of the current preset convolution layer to perform two-dimensional convolution processing to obtain two-dimensional convolution features; performing normalization processing on the two-dimensional convolution features to obtain normalized features; performing nonlinear processing on the normalized features to obtain processed features, and taking the processed features as the features output by the current preset convolution layer.

[0010] In some embodiments, the target loss is determined based on a first difference between the first prediction result of each sample image and the annotation result in each sample label, and a second difference between the second prediction result of each sample image and the sample identification in each sample label, including: determining a first loss based on the first difference; determining a second loss based on the second difference; and performing weighted summation on the first loss and the second loss to obtain the target loss.

[0011] In some embodiments, the training method of the image processing model further includes: obtaining a to-be-detected image; and performing image processing on the to-be-detected image by using the trained image processing model to obtain an image processing result of the to-be-detected image.

[0012] The present application provides a training device of an image processing model, including: an obtaining module, an image processing module, a determining module, and an adjusting module; the obtaining module is configured to obtain sample images related to a plurality of perception tasks of a vehicle and sample labels corresponding to each sample image, wherein the sample label of each sample image includes a sample identification of a perception task to which the sample image belongs and an annotation result related to the perception task in the sample image; the image processing module is configured to perform image processing on each sample image by using an image processing model to obtain a first prediction result and a second prediction result of each sample image, wherein the first prediction result of each sample image is used to represent a prediction result of the sample image about at least one perception task, and the second prediction result of each sample image is used to represent the perception task to which the sample image belongs; the determining module is configured to determine a target loss based on a first difference between the first prediction result of each sample image and the annotation result in each sample label, and a second difference between the second prediction result of each sample image and the sample identification in each sample label; and the adjusting module is configured to adjust parameters in the image processing model by using the target loss to obtain a trained image processing model.

[0013] The present application provides an electronic device including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the training method of the image processing model.

[0014] The present application provides a computer-readable storage medium having program instructions stored thereon, wherein the program instructions are executed by a processor to implement the training method of the image processing model.

[0015] The scheme uses an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, uses a first difference between the first prediction result of each sample image with respect to at least one perception task and a label result in each sample label and a second difference between the second prediction result of each sample image for the perception task to which the sample image belongs and a sample identifier in each sample label, adjusts parameters in the obtained target loss image processing model, and obtains a trained image processing model. Compared with the prior art which only uses the first difference to determine the target loss, the target loss obtained by combining the first difference and the second difference can better fit the loss required by each parameter in the image processing model, and the prediction performance of the trained image processing model obtained by using the target loss is better.

[0016] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, rather than limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0018] Figure 1 is a flowchart of an embodiment of the training method of the image processing model of the present application Figure 1 ;

[0019] Figure 2 is a flowchart of an embodiment of the training method of the image processing model of the present application Figure 2 ;

[0020] Figure 3 is a flowchart of an embodiment of the training method of the image processing model of the present application Figure 3 ;

[0021] Figure 4 is a schematic diagram of the image processing model in an embodiment of the training method of the image processing model of the present application

[0022] Figure 5 is a structural schematic diagram of an embodiment of the training device of the image processing model of the present application

[0023] Figure 6 is a structural schematic diagram of an embodiment of the electronic device of the present application

[0024] Figure 7 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application DETAILED DESCRIPTION

[0025] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0026] In the following description, specific details are set forth in order to provide a thorough understanding of the present application. The present application can be practiced without resorting to the details specific.

[0027] The term "and / or", used herein only describes an associated relationship for associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects. In addition, "multiple" herein means two or more than two. In addition, the term "at least one" herein means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B and C, which can mean including any one or more elements selected from the set consisting of A, B and C.

[0028] The present application provides a training method of an image processing model and a training device of the image processing model. The application scenario of the training method of the image processing model includes but is not limited to the training process of a detection model related to a vehicle perception task. The execution subject of the training method of the image processing model can be the training device of the image processing model. For example, the training device of the image processing model can be arranged in a terminal device or a server or other processing device, wherein the terminal device can be a device for training of the image processing model, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, etc. In some possible implementation manners, the training method of the image processing model can be realized by a processor calling computer readable instructions stored in a memory.

[0029] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the training method of the image processing model of the present application Figure 1 . Specifically, the training method of the image processing model can include the following steps:

[0030] Step S11: obtaining sample images related to several kinds of perception tasks of vehicles and sample labels corresponding to each sample image.

[0031] The sample label of each sample image includes a sample identification of the perception task to which the sample image belongs and a labeling result related to the perception task in the sample image.

[0032] The several perception tasks of the vehicle can be image detection related tasks in a traffic scene related to the vehicle. The several perception tasks can be one or more. Each perception task can be a preset detection on an image collected by an image collection device of the vehicle. For example, the perception task can be a dynamic obstacle detection, a static obstacle detection, a traffic light recognition, a traffic sign recognition, a lane line ground mark detection, a scene detection in which the vehicle is located, a drivable area segmentation, and a detection on whether the image collection device of the vehicle is blocked, and the like, which are all types of tasks related to vehicle perception. The present application takes three perception tasks as an example, and the several perception tasks include a perception task A, a perception task B, and a perception task C. This is only an example and does not limit the types and quantities of the several perception tasks.

[0033] The sample images can be training image data obtained by pre-labeling images collected by vehicles in traffic scenes. Different sample images can be training image data corresponding to different perception tasks. The sample labels corresponding to the sample images can include sample identifiers corresponding to the sample images and labeling results of the sample images. Specifically, the sample labels corresponding to the sample images can be used to represent sample identifiers related to at least one perception task and / or the sample images belong to training image data of at least one perception task obtained by labeling the sample images, for example, a certain sample image is used to train the image processing model for perception task A, and the sample identifier of the sample image can be perception task A, or a certain sample image is used to train the image processing model for perception tasks B and C, and the sample identifier of the sample image can be perception tasks B and C. For example, in the case of perception task A being a vehicle scene detection task, the labeling result of the sample image belonging to perception task A can be traffic scene one, traffic scene two, traffic scene three, etc. The labeling result corresponding to the sample image can be a label that the sample image belongs to a traffic scene, which can be a traffic scene category such as a vehicle in a tunnel scene, a vehicle in an elevated scene, and a vehicle in an urban scene. The sample identifier of the sample image can be the type of perception task to which the sample image belongs, which can be whether the sample image is training image data of perception task A. For example, in the case of perception task B being a drivable area segmentation task, the labeling result of the sample image belonging to perception task B can indicate whether the image region of the sample image includes a road surface or not. The labeling result corresponding to the sample image can be a label that the image region of the sample image belongs to a road surface or not, which can be that if at least part of the image region in the image belongs to a road surface, the image region is a drivable region, otherwise the image region is a non-drivable region. The sample identifier of the sample image can be the type of perception task to which the sample image belongs, which can be whether the sample image is training image data of perception task B. For example, in the case of perception task C being a vehicle image acquisition device occlusion detection task, the labeling result of the sample image belonging to perception task C can be that the sample image is collected in the case that the image acquisition device is occluded, or that the sample image is collected in the case that the image acquisition device is not occluded. The labeling result corresponding to the sample image can be a label that the image region of the sample image is occluded by a specific object, which can be that if at least part of the image region in the image is occluded by a specific object, the image acquisition device collecting the sample image is occluded, otherwise the image acquisition device collecting the sample image is not occluded. For example, the specific object can be an animal, mud, stains, etc. The sample identifier of the sample image can be the type of perception task to which the sample image belongs, which can be whether the sample image is training image data of perception task C.

[0034] The way of obtaining each sample image and the sample label corresponding to each sample image can be obtaining training image data related to the vehicle perception task from a public data set, and can also be directly obtained from a preset database. Specifically, the image collected by the vehicle is taken as a sample image, and the sample label is obtained by automatically pre-labeling each sample image and stored in the preset database.

[0035] Step S12: using the image processing model to perform image processing on each sample image to obtain the first prediction result and the second prediction result of each sample image.

[0036] The first prediction result of each sample image is used to represent the prediction result of the sample image on at least one perception task, and the second prediction result of each sample image is used to represent the perception task to which the sample image belongs.

[0037] In some application scenarios, image processing can be at least one perception task related image processing operation on the sample image. According to different perception tasks, the image processing operation in the image processing model is different. Specifically, the image processing corresponding to the image processing model can be feature extraction on the sample image first, and then processing operation matching the perception task on the image after feature extraction. It can be understood that in the case of different perception tasks, all sample images are input into the same preset feature extraction network for feature extraction. The preset feature extraction network in the image processing model can be a MobileNet feature extraction network, an Inception feature extraction network, etc. The setting method of the feature extraction network is not limited herein. In some application scenarios, all sample images are input into the same preset feature extraction network for feature extraction, and all perception task matching processing operations are performed on all sample images after feature extraction to obtain the first prediction result of all sample images. It can be understood that each sample image after feature extraction will perform all perception task matching image processing operations. In other application scenarios, all sample images are input into the same preset feature extraction network for feature extraction, and only the perception task matching processing operation corresponding to the sample image is performed on all sample images after feature extraction to obtain the first prediction result of each sample image. It can be understood that each sample image will only perform the perception task matching image processing operation corresponding to the sample label of the sample image. Exemplarily, for the sample image of the perception task A, the first prediction result of the sample image is obtained by performing the processing operation belonging to the perception task A matching operation on the sample image after feature extraction. Wherein, the processing operation belonging to the perception task A matching operation can be vehicle scene detection on the sample image after feature extraction.

[0038] In some other application scenarios, the image processing can be image classification processing on the sample image to obtain a second prediction result of the sample image. Specifically, the image processing corresponding to the image processing model can be feature extraction on the sample image first, and then classification processing on the image after the feature extraction to obtain the second prediction result of the sample image. The second prediction result of the sample image can be training image data indicating that the sample image belongs to at least one of the perception tasks. For example, for the sample image of the perception task A, the classification processing on the sample image after the feature extraction obtains the second prediction result of the sample image. For different sample images, the same classification processing is performed on each sample image after the feature extraction to obtain the second prediction result of each sample image. For the sample image of the perception task A, the second prediction result of the sample image is used to indicate whether the sample image belongs to the training image data of the perception task A.

[0039] Step S13: determining a target loss based on the first difference between the first prediction result of each sample image and the annotation result in each sample label, and the second difference between the second prediction result of each sample image and the sample identification in each sample label.

[0040] The first difference can be used to evaluate the performance of the image processing model on a specific perception task. The second difference can be used to evaluate the performance of the image processing model on the perception task classification. Specifically, the second difference can be used to evaluate the performance of the shared layer in the image processing model on the perception task classification. For example, the shared layer in the image processing model is used to represent a preset feature extraction network used by the sample image regardless of the category of the perception task to which the sample image belongs. The target loss can be an overall loss function used to guide the optimization of the image processing model in combination with the first difference and the second difference.

[0041] Step S14: adjusting the parameters in the image processing model using the target loss to obtain a trained image processing model.

[0042] The parameters in the image processing model can be the parameters of the network or module included in the image processing model. The optimization of the image processing model using the target loss can be adjusting the parameters in the image processing model using gradient descent method or adaptive learning rate adjustment, training the image processing model with the target loss being smaller as the goal until the target loss output by the trained image processing model meets the requirements. The gradient descent method can be, for example, stochastic gradient descent (SGD), Adam, and the like, which adjusts the model parameters through back propagation. The adaptive learning rate adjustment can be dynamically adjusting the target loss according to the training process.

[0043] It can be considered that the target loss combined with the first difference and the second difference used in the present application can better fit the loss required by each parameter in the image processing model, and can additionally train the performance of the shared layer in the image processing model on the perception task classification, so that the sample image after feature extraction output by the shared layer in the image processing model can adapt to the image processing operation matched by multiple perception tasks, that is, the sample image after feature extraction output by the trained shared layer in the image processing model is a feature with smaller discrimination between different perception tasks, so as to improve the accuracy of the first prediction result obtained by the sample image after feature extraction in the subsequent image processing operation matched by different perception tasks. Specifically, using the second loss can make the output features of the shared layer in the image processing model as similar as possible between multiple perception tasks, so as to finally achieve that the shared part in the image processing model can generate input features beneficial to the subsequent image processing network corresponding to different perception tasks.

[0044] The above scheme uses the image processing model to perform image processing on each sample image to obtain the first prediction result and the second prediction result of each sample image, uses the first difference between the first prediction result of each sample image with respect to at least one perception task and the annotation result in each sample label, and the second difference between the second prediction result of each sample image and the sample identification in each sample label, determines the target loss obtained to adjust the parameters in the image processing model, and obtains the trained image processing model. Compared with the prior art which only uses the first difference to determine the target loss, the target loss combined with the first difference and the second difference used in the present application can better fit the loss required by each parameter in the image processing model, and then the prediction performance of the trained image processing model obtained using the target loss is better.

[0045] Please refer to Figure 2 , Figure 2 is a flowchart of an embodiment of the training method of the image processing model of the present application Figure 2 .

[0046] In some embodiments, the above step S12 can include the following steps: step S21: performing feature extraction on each sample image to obtain sample features corresponding to each sample image. Step S22: for each sample image, input the sample features corresponding to the sample image into the image processing network related to the perception task to which the sample image belongs to obtain the first prediction result corresponding to the sample image. Step S23: respectively input the sample features corresponding to each sample image into the preset task classifier to obtain the second prediction result corresponding to each sample image.

[0047] The image processing model can include a preset feature extraction network, a plurality of image processing networks with the same number of categories as the number of perception tasks, and a preset task classifier. Different perception tasks match different image processing networks. For each sample image, the sample feature corresponding to the sample image is input into the image processing network matched with the perception task to which the sample image belongs to perform the image processing operation matched with each perception task to obtain a first prediction result corresponding to the sample image. For example, the image processing network can be a perception task A network matched with the perception task A, specifically a vehicle scene detection network. The sample feature of each sample image is input into the perception task A network to perform the image processing operation matched with the perception task A to obtain a first prediction result. The first prediction result can represent the traffic scene to which the sample image belongs and the probability value of the traffic scene. The traffic scene with the largest probability value among the probability values of the traffic scenes is determined as the traffic scene to which the sample image belongs. The probability value of each traffic scene can represent the possibility of the sample image belonging to each traffic scene.

[0048] For example, the image processing network can be a perception task B network matched with the perception task B, specifically a drivable area segmentation network. The sample feature of each sample image is input into the perception task B network to perform the image processing operation matched with the perception task B to obtain a first prediction result. The first prediction result can represent whether at least part of the image region is a drivable area or a non-drivable area. Specifically, the first prediction result includes whether at least part of the image region is a drivable area and the probability value corresponding to each image region. Each detection box obtained by target detection on the sample image can be an image region contained in the sample image, and the probability value corresponding to each image region represents the possibility of the image region belonging to the drivable area.

[0049] For example, the image processing network can be a perception task C network matched with the perception task C, specifically a vehicle image acquisition device occlusion detection network. The sample feature of each sample image is input into the perception task C network to perform the image processing operation matched with the perception task C to obtain a first prediction result. The first prediction result can represent whether the image region of the sample image is occluded by a specific object. Specifically, the first prediction result includes the probability value of the sample image being occluded by each specific object, and the probability value of each specific object occlusion represents the possibility of the sample image being occluded by each specific object. The specific object with the largest probability value among the probability values of the specific object occlusion is determined as the specific object occluding the image acquisition device in the first prediction result.

[0050] The step S23 can be that the sample features of the sample images are input into the preset task classifier in batches for classification processing to obtain the second prediction results of the sample images. The preset task classifier can be a preset multi-task classification network. Specifically, the sample features of the sample images are directly input into a fully connected layer to obtain the output of the fully connected layer. The output of the fully connected layer is processed by an activation function to convert the output into a probability distribution to obtain the probability values of each category. Each category can be a category of the sample images belonging to each perception task. The category with the maximum probability value among the probability values of each category is determined as the perception task to which the sample image belongs.

[0051] The preset feature extraction network can perform the step S21. After the step S21, each image processing network can perform the step S22. After the step S21, the preset task classifier can perform the step S23. The step S22 and the step S23 can be performed in parallel.

[0052] It can be considered that each image processing network can focus on its specific perception task without being disturbed by other perception tasks. For each sample image, the sample feature corresponding to the sample image is only input into the image processing network matched with the perception task to obtain the first prediction result corresponding to the sample image. The first prediction result is used to determine the target loss to train the image processing model, so that the image processing model can better learn the features related to the specific perception task, thereby improving the performance of the trained image processing model in the image processing operation of the image processing network corresponding to the different perception tasks.

[0053] Please refer to Figure 3 , Figure 3 is a flowchart of an embodiment of the method for training the image processing model of the present application Figure 3 .

[0054] In some embodiments, the step S23 can include the following steps performed for each sample image as shown in Figure 4 .

[0055] Step S31: performing convolution processing on the sample features of the sample images to obtain convolution features of the sample images.

[0056] The preset task classifier comprises a preset convolution module. The sample feature of the sample image is input into the preset convolution module for convolution processing to obtain the convolution feature of the sample image. The preset convolution module can comprise a plurality of preset convolution layers arranged in cascade. The structure and parameters of each preset convolution layer can be the same or different. Each preset convolution layer can correspond to a convolution processing mode. Specifically, the convolution processing mode can be any one or at least one of 3D convolution processing, 1x1 convolution processing, transpose convolution processing and point-by-point grouping convolution. It can be understood that the arrangement of the plurality of preset convolution layers in the preset convolution module is not limited.

[0057] In some embodiments, the step S31 is performed by a preset convolution module in the image processing model. The preset convolution module comprises a plurality of preset convolution layers arranged in cascade. The number of the plurality of preset convolution layers is the same as the number of the perception tasks. The feature output by the last preset convolution layer is taken as the convolution feature of the sample image. The preset convolution module performs the above-mentioned manner of inputting the sample feature of the sample image into the preset convolution module for convolution processing to obtain the convolution feature of the sample image, which comprises: taking the sample feature of the sample image or the feature output by the last preset convolution layer as the input of the current preset convolution layer for two-dimensional convolution processing to obtain a two-dimensional convolution feature. The two-dimensional convolution feature is normalized to obtain a normalized feature. The normalized feature is subjected to nonlinear processing to obtain a processed feature, and the processed feature is taken as the feature output by the current preset convolution layer.

[0058] The structure of each preset convolution layer in the plurality of preset convolution layers arranged in cascade is the same, but the parameters can be the same or different. The current preset convolution layer can be any one of the preset convolution layers in the preset convolution module. For example, the current preset convolution layer can be the first preset convolution layer or a preset convolution layer other than the first preset convolution layer.

[0059] For example, when the current preset convolution layer is the first preset convolution layer, the sample feature of the sample image is taken as the input of the current preset convolution layer for two-dimensional convolution processing to obtain a two-dimensional convolution feature. The sample feature of the sample image is input into the first preset convolution layer for two-dimensional convolution processing to obtain a two-dimensional convolution feature of the first preset convolution layer. The two-dimensional convolution feature of the first preset convolution layer is normalized to obtain a normalized feature of the first preset convolution layer. For example, the normalization processing can be represented as BN processing. The normalized feature is subjected to nonlinear processing to obtain a processed feature, and the processed feature is taken as the feature output by the first preset convolution layer. The nonlinear processing can be nonlinear transformation of the normalized feature of the first preset convolution layer using a preset activation function to obtain the processed feature. The preset activation function can be set as a ReLU activation function, a Sigmoid activation function, a Tanh activation function, etc.

[0060] Exemplarily, in a case that the current preset convolutional layer is a non-first preset convolutional layer, for example, the current preset convolutional layer is the last preset convolutional layer, the sample feature of the sample image is subjected to two-dimensional convolution processing as an input of the current preset convolutional layer, to obtain a two-dimensional convolution feature. The sample feature of the sample image is input into the last preset convolutional layer for two-dimensional convolution processing, to obtain a two-dimensional convolution feature of the last preset convolutional layer. The two-dimensional convolution feature of the last preset convolutional layer is subjected to normalization processing, to obtain a normalized feature of the last preset convolutional layer. The normalized feature is subjected to nonlinear processing, to obtain a processed feature, and the processed feature is taken as the feature output by the last preset convolutional layer. The nonlinear processing can be nonlinear transformation of the normalized feature of the last preset convolutional layer using a preset activation function to obtain the processed feature.

[0061] It can be considered that, according to the number of categories of perception tasks, the preset convolutional layers in the preset convolutional module are set, which can enable the image processing model to enhance important features in the sample image for each sample feature, thereby improving the accuracy of determining whether the sample image belongs to image data of any kind of perception task.

[0062] Step S32: performing dimension transformation processing on the convolution feature of the sample image, to obtain a transformed feature corresponding to the sample image.

[0063] The preset task classifier includes a preset dimension transformation module. The preset dimension transformation module includes preset dimension transformation layers arranged in cascade.

[0064] The convolution feature of the sample image is input into the preset dimension transformation module for dimension transformation processing, to obtain a transformed feature corresponding to the sample image. The structure and parameters of each preset dimension transformation layer can be the same or different. Each preset dimension transformation layer can correspond to a dimension transformation mode. Exemplarily, the preset dimension transformation layer can be a linear layer. The preset dimension transformation module is used to convert the convolution feature to a target dimension, so as to perform a classification task. Specifically, the preset dimension transformation layer can be a fully connected layer (Fully Connected Layer), a linear layer (Linear Layer), and a multilayer perceptron MLP layer (Multilayer Perceptron, MLP). It can be understood that the arrangement mode of the preset dimension transformation layers in the preset dimension transformation module is not limited here.

[0065] In some embodiments, the step S32 can include the following steps: firstly, performing a global average pooling operation on the convolutional features of the sample image to obtain the pooled features corresponding to the sample image; then, performing a first dimension transformation processing on the pooled features to obtain the first dimension transformed features corresponding to the sample image; and then, performing a second dimension transformation processing on the first dimension transformed features corresponding to the sample image to obtain the second dimension transformed features corresponding to the sample image, and taking the second dimension transformed features corresponding to the sample image as the transformed features corresponding to the sample image.

[0066] Exemplarily, the input of the task classifier is the sample feature extracted by the preset feature extraction network in the shared layer from the sample image. The dimension of the sample feature is (N, C, H, W), where N is the mini-batch size, C is the number of feature channels, and H and W are the height and width, respectively. The dimension of the convolutional feature obtained after the sample feature passes through the preset convolutional module does not change. The preset dimension transformation module includes a first linear layer and a second linear layer. Both the first linear layer and the second linear layer can be represented as a linear layer. For the first linear layer, the number of input channels is C, and the number of output channels is 2048. For the second linear layer, the number of input and output channels is 2048, and the number of output channels is the number of categories of the perception task. If the number of categories of the perception task is 4, the number of output channels is 4. If the number of categories of the perception task is 3, the number of output channels is 3. The preset dimension transformation module can also include a pooling layer and a feature transformation layer. The pooling layer can be used to perform a global average pooling processing on the input features of the pooling layer. The feature transformation layer can use the view function to perform feature transformation on the input features to obtain the output of the feature transformation layer. The first dimension transformed features corresponding to the sample image are obtained by performing a first dimension transformation processing on the pooled features input into the feature transformation layer. The second dimension transformed features corresponding to the sample image are obtained by performing a second dimension transformation processing on the first dimension transformed features corresponding to the sample image input into the first linear layer and the second linear layer.

[0067] Step S33: determining the second prediction result corresponding to the sample image based on the transformed features corresponding to the sample image.

[0068] The transformed features corresponding to each sample image are processed by an activation function to convert the output into a probability distribution to obtain the probability values of each category. Each category can be a category of the perception task to which the sample image belongs. The category with the largest probability value among the probability values of each category is determined as the perception task to which the sample image belongs. Specifically, the category with the largest probability value among the probability values of each category is determined as the second prediction result of the sample image. The preset activation function can be set as a ReLU activation function, a Sigmoid activation function, a Tanh activation function, etc.

[0069] It can be considered that the accuracy of the determined second prediction result is improved by classifying the belonging of each sample feature to a perception task through the preset task classifier, so that the parameters of the shared layer in the image processing model trained using the target loss are more in line with the training requirements, and thus the prediction performance of the trained image processing model is improved.

[0070] In some embodiments, the step S13 can include the following steps: determining a first loss based on the first difference; determining a second loss based on the second difference; and performing weighted summation on the first loss and the second loss to obtain the target loss.

[0071] Specifically, the manner of calculating the target loss can refer to formula (1): TL = w1 x loss1 + … + wn x lossn + w0 x (L1 + … + Ln) formula (1);

[0072] wherein, TL can be used to represent the target loss. lossn can represent the first difference obtained by the sample image belonging to the nth perception task, and the nth first loss determined. wn can be used to represent the weight of the n first losses. loss1 can represent the first difference obtained by the sample image belonging to any one perception task, and any one first loss determined. w1 can be used to represent the weight of any one first loss. Ln can represent the second difference obtained by the sample image belonging to the nth perception task, and the nth second loss determined. L1 can represent the second difference obtained by the sample image belonging to any one perception task, and any one second loss determined. w0 can represent the weight corresponding to all second losses.

[0073] Exemplarily, for each sample image, the sample identifier of the sample image belonging to the perception task is in the form of One-hot encoding. For example, if the sample image is training image data belonging to perception task A, the sample identifier of the sample image is set to the first encoding result (1, 0, 0). If the sample image is training image data belonging to perception task B, the sample identifier of the sample image is set to the second encoding result (0, 1, 0). If the sample image is training image data belonging to perception task C, the sample identifier of the sample image is set to the third encoding result (0, 0, 1). The present application can set the mini-batch to 1, that is, the output dimension of the task classifier is (1, 3). Assuming that the sample image belonging to perception task A is subjected to feature extraction by the preset feature extraction network, and the second prediction result of the output of the preset task classifier is (0.8, 0.1, 0.2). The second loss of the sample image can be obtained by calculating the loss function between the second prediction result (0.8, 0.1, 0.2) and the first encoding result (1, 0, 0). It can be understood that the second loss of the sample image belonging to other perception tasks is analogously obtained.

[0074] In some application scenarios, the cross-entropy loss function is used to define the second loss corresponding to the preset task classifier. Specifically, the process of determining the second loss based on the second difference can refer to formula (2):

[0075]

[0076] wherein C is the number of task categories, which can be the number of categories of a plurality of perception tasks, that is, the number of perception tasks. c and x i may represent the output result of the preset task classifier when the task category is i and c, respectively. Similarly, y c may represent the task identifier corresponding to the task category c. For example, y c may be used to represent the sample identifier of any sample image when the task category is c. loss can be used to represent the second loss of any sample image. y c may be used to represent the sample identifier of any sample image. The sample identifier is usually obtained in the form of one-hot encoding.

[0077] In some embodiments, the image processing model training method further comprises: obtaining a to-be-detected image. The trained image processing model is used to process the to-be-detected image to obtain an image processing result of the to-be-detected image.

[0078] The to-be-detected image can be an image that needs to perform image processing related to the above-mentioned several perception tasks. The to-be-detected image can be an image collected by an image collection device of a vehicle. The image processing result of the to-be-detected image can be a prediction result of a perception task to which the to-be-detected image belongs, obtained by performing image processing related to the perception task on the to-be-detected image.

[0079] The to-be-detected image is input into the trained image processing model for image processing to obtain an image processing result of the to-be-detected image. Specifically, the to-be-detected image is input into the preset feature extraction network in the trained image processing model for feature extraction to obtain a to-be-detected image after feature extraction. The to-be-detected image after feature extraction is input into the image processing network matched with the perception task in the image processing model for image processing to obtain a prediction result of the to-be-detected image. It can be understood that the above-mentioned inputting the to-be-detected image into the preset feature extraction network in the trained image processing model for feature extraction to obtain the to-be-detected image after feature extraction can refer to the above-mentioned step S21, which will not be repeated here. The above-mentioned inputting the to-be-detected image after feature extraction into the image processing network matched with the perception task in the image processing model for image processing to obtain the prediction result of the to-be-detected image can refer to the above-mentioned step S22, which will not be repeated here.

[0080] In some application scenarios, the to-be-detected image and the perception task matched with the to-be-detected image are obtained. The perception task matched with the to-be-detected image can be obtained by analyzing an operation instruction on a display interface of the vehicle. The to-be-detected image is processed by using the trained image processing model to obtain an image processing result of the to-be-detected image. The to-be-detected image after feature extraction is input into the image processing network matched with the perception task matched with the to-be-detected image in the image processing model for image processing to obtain the prediction result of the to-be-detected image.

[0081] Please refer to Figure 4 , Figure 5 is a schematic diagram of an image processing model in an embodiment of the training method of the image processing model of the present application.

[0082] The image processing model comprises a shared layer, image processing networks matched with respective perception tasks, and a preset task classifier. The shared layer can be the preset feature extraction network described above. The image processing networks matched with respective perception tasks can be the perception task A network, the perception task B network, and the perception task C network described above. The preset task classifier comprises a preset convolution module and a preset dimension transformation module arranged in cascade. For the arrangement of network and module structure and the functions performed in the preset convolution module and the preset dimension transformation module, reference can be made to the above description, which will not be repeated here. It can be understood that the image processing networks matched with respective perception tasks in the image processing model are at least two image processing networks, which are only used for illustration and do not limit the number of perception tasks in the image processing model.

[0083] For example, the sample image A, the sample image B, and the sample image C can be training image data respectively identified as belonging to the perception task A, the perception task B, and the perception task C. Each of the sample image A, the sample image B, and the sample image C can comprise one or more sample images. For example, taking the sample image A as one sample image, the sample image A is input into the shared layer, and step S21 is performed to obtain the sample feature of the sample image A output by the shared layer. In some application scenarios, the sample feature of the sample image A is input into the perception task A network described above, and step S22 is performed to obtain the first prediction result corresponding to the sample image A. Then, the first loss is determined based on the first difference between the first prediction result of each sample image and the annotation result in each sample label. In other application scenarios, the sample feature of the sample image A is input into the preset task classifier described above, and step S23 is performed to obtain the second prediction result corresponding to the sample image A. Specifically, the preset dimension transformation module in the preset task classifier comprises a pooling layer, a feature transformation layer, a first linear layer, and a second linear layer. For the pooling layer, the feature transformation layer, the first linear layer, and the second linear layer, reference can be made to the above description, which will not be repeated here. The convolution feature output by the preset convolution module is sequentially input into the pooling layer, the feature transformation layer, the first linear layer, and the second linear layer to obtain the second dimension-transformed feature output by the second linear layer. The second dimension-transformed feature is taken as the transformed feature corresponding to the sample image A. Then, the second loss is determined based on the second difference between the second prediction result of each sample image and the sample identification in each sample label. The processing of the sample image B and the sample image C can be understood with reference to the above description, which will not be repeated here.

[0084] The first loss of all sample images and the second loss of all sample images are weighted and summed to obtain a target loss. The target loss is used to adjust at least part of the modules or network layers in the image processing model through back propagation. For example, only the parameters in the image processing model including the shared layer, the image processing network matched with each perception task, and the preset part of the preset task classifier are adjusted. The preset part of the preset task classifier can be all layers except the pooling layer in the preset convolution module and the preset dimension transformation module.

[0085] The above scheme uses the image processing model to perform image processing on each sample image to obtain the first prediction result and the second prediction result of each sample image. The first difference between the first prediction result of each sample image with respect to at least one perception task and the annotation result in each sample label, and the second difference between the second prediction result of each sample image with respect to the perception task to which the sample image belongs and the sample identifier in each sample label are used to determine the target loss, adjust the parameters in the image processing model, and obtain the trained image processing model. Compared with the prior art which only uses the first difference to determine the target loss, the target loss obtained by combining the first difference and the second difference can better fit the loss required by each parameter in the image processing model, and the prediction performance of the trained image processing model obtained by using the target loss is better.

[0086] Referring to Figure 5 , Figure 6 is a structural schematic diagram of an embodiment of the image processing model training apparatus of the present application. The image processing model training apparatus 50 includes an acquisition module 51, an image processing module 52, a determination module 53, and an adjustment module 54. The acquisition module 51 is configured to acquire sample images related to several perception tasks of a vehicle and sample labels corresponding to each sample image. The sample label of each sample image includes a sample identifier of the perception task to which the sample image belongs and an annotation result related to the perception task in the sample image. The image processing module 52 is configured to use an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image. The first prediction result of each sample image is used to represent the prediction result of the sample image with respect to at least one perception task, and the second prediction result of each sample image is used to represent the perception task to which the sample image belongs. The determination module 53 is configured to determine a target loss based on the first difference between the first prediction result of each sample image and the annotation result in each sample label, and the second difference between the second prediction result of each sample image and the sample identifier in each sample label. The adjustment module 54 is configured to use the target loss to adjust the parameters in the image processing model to obtain a trained image processing model.

[0087] The scheme uses an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, uses a first difference between the first prediction result of each sample image with respect to at least one perception task and a label result in each sample label and a second difference between a second prediction result of each sample image for a perception task to which the sample image belongs and a sample identifier in each sample label, adjusts parameters in the obtained image processing model based on the target loss, and obtains a trained image processing model. Compared with the prior art which only uses the first difference to determine the target loss, the target loss obtained by combining the first difference and the second difference can better fit the loss required by each parameter in the image processing model, and the prediction performance of the trained image processing model obtained by using the target loss is better.

[0088] The functions performed by each module can refer to the training method of the image processing model, which will not be described here.

[0089] Please refer to Figure 6 , Figure 7 is a structural schematic diagram of an embodiment of the vehicle. The electronic device 60 includes a memory 61 and a processor 62, and the processor 62 is configured to execute program instructions stored in the memory 61 to implement the steps in the above-mentioned embodiment of the training method of the image processing model. In a specific implementation scenario, the electronic device 60 can include but is not limited to a microcomputer, a server, and in addition, the electronic device 60 can also include a notebook computer, a tablet computer, and other mobile devices, which are not limited here.

[0090] Specifically, the processor 62 is configured to control itself and the memory 61 to implement the steps in the above-mentioned embodiment of the training method of the image processing model. The processor 62 can also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 62 can be an integrated circuit chip with signal processing capability. The processor 62 can also be a general-purpose processor, a DSP (Digital Signal Processor, digital signal processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field-Programmable Gate Array, field programmable gate array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 62 can be implemented by an integrated circuit chip together.

[0091] The scheme uses an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, uses a first difference between the first prediction result of each sample image on at least one perception task and a labeled result in each sample label and a second difference between the second prediction result of each sample image on the perception task to which the sample image belongs and a sample identifier in each sample label to determine a parameter in the image processing model obtained by the target loss, and obtains a trained image processing model. Compared with the prior art which only uses the first difference to determine the target loss, the target loss obtained by combining the first difference and the second difference can better fit the loss required by each parameter in the image processing model, and the prediction performance of the trained image processing model obtained by using the target loss is better.

[0092] Please refer to Figure 7 , ​ is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. The computer readable storage medium 70 stores program instructions 701, and the program instructions 701 are executed by a processor to implement the steps in any of the above image processing model training method embodiments.

[0093] The scheme uses an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, uses a first difference between the first prediction result of each sample image on at least one perception task and a labeled result in each sample label and a second difference between the second prediction result of each sample image on the perception task to which the sample image belongs and a sample identifier in each sample label to determine a parameter in the image processing model obtained by the target loss, and obtains a trained image processing model. Compared with the prior art which only uses the first difference to determine the target loss, the target loss obtained by combining the first difference and the second difference can better fit the loss required by each parameter in the image processing model, and the prediction performance of the trained image processing model obtained by using the target loss is better.

[0094] In some embodiments, the system provided by the embodiments of the present disclosure has functions or contains modules that can be used to execute the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, they will not be repeated here.

[0095] The above description of each embodiment tends to emphasize the differences between each embodiment, and the same or similar parts can be mutually referred to. For brevity, they will not be repeated here.

[0096] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other manners. For example, the division of the apparatus embodiments described above is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0097] In addition, each function unit in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of a software function unit.

[0098] If the integrated unit is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially or substantially, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various other media that can store program codes.

Claims

1. A method for training an image processing model, characterized in that, The method comprises: obtaining sample images related to a plurality of perception tasks of a vehicle and sample labels corresponding to the sample images, wherein each sample label of a sample image comprises a sample identification of a perception task to which the sample image belongs and a labeling result related to the perception task in the sample image; performing image processing on each sample image by using an image processing model to obtain a first prediction result and a second prediction result of each sample image, comprising: performing feature extraction on each sample image to obtain a sample feature corresponding to each sample image; for each sample image, inputting the sample feature corresponding to the sample image into an image processing network related to the perception task to which the sample image belongs to obtain a first prediction result corresponding to the sample image; and inputting the sample feature corresponding to each sample image into a preset task classifier to obtain a second prediction result corresponding to each sample image; wherein the first prediction result of each sample image is used to represent a prediction result of the sample image on at least one of the perception tasks, and the second prediction result of each sample image is used to represent the perception task to which the sample image belongs; determining a target loss based on a first difference between the first prediction result of each sample image and the labeling result in each sample label, and a second difference between the second prediction result of each sample image and the sample identification in each sample label; adjusting parameters in the image processing model by using the target loss to obtain a trained image processing model.

2. The method of claim 1, wherein, The step of inputting the sample feature corresponding to each sample image into the preset task classifier to obtain the second prediction result corresponding to each sample image comprises: for each sample image, performing the following steps: performing convolution processing on the sample feature of the sample image to obtain a convolution feature of the sample image; performing dimension transformation processing on the convolution feature of the sample image to obtain a transformed feature corresponding to the sample image; determining the second prediction result corresponding to the sample image based on the transformed feature corresponding to the sample image.

3. The method of claim 2, wherein, The step of performing dimension transformation processing on the convolution feature of the sample image to obtain the transformed feature corresponding to the sample image comprises: performing global average pooling operation on the convolution feature of the sample image to obtain a pooled feature corresponding to the sample image; performing first dimension transformation processing on the pooled feature to obtain a first dimension transformed feature corresponding to the sample image; performing second dimension transformation processing on the first dimension transformed feature corresponding to the sample image to obtain a second dimension transformed feature corresponding to the sample image, and taking the second dimension transformed feature corresponding to the sample image as the transformed feature corresponding to the sample image.

4. The method of claim 2, wherein, The step of performing convolution processing on the sample features of the sample image to obtain the convolution features of the sample image is performed by a preset convolution module in the image processing model, the preset convolution module includes a plurality of preset convolution layers arranged in cascade, the number of the plurality of preset convolution layers is the same as the number of perception tasks, and the features output by the last preset convolution layer are taken as the convolution features of the sample image. The manner in which the preset convolution module performs convolution processing on the sample feature input of the sample image to obtain the convolution features of the sample image comprises the following steps: performing two-dimensional convolution processing on the sample features of the sample image or the features output by the last preset convolution layer as the input of the current preset convolution layer to obtain two-dimensional convolution features; performing normalization processing on the two-dimensional convolution features to obtain normalized features; performing nonlinear processing on the normalized features to obtain processed features, and taking the processed features as the features output by the current preset convolution layer.

5. The method according to any one of claims 1 to 4, characterized in that, The target loss is determined based on a first difference between the first prediction result of each sample image and the labeled result in each sample label, and a second difference between the second prediction result of each sample image and the sample identification in each sample label, and comprises the following steps: determining a first loss based on the first difference; determining a second loss based on the second difference; performing weighted summation on the first loss and the second loss to obtain the target loss.

6. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: obtaining a to-be-detected image; performing image processing on the to-be-detected image by using the trained image processing model to obtain an image processing result of the to-be-detected image.

7. A training device for an image processing model, characterized in that, Comprise: an acquisition module configured to acquire sample images related to a plurality of perception tasks of a vehicle and sample labels corresponding to the sample images, wherein each sample label of each sample image comprises a sample identification of a perception task to which the sample image belongs and a labeled result related to the perception task in the sample image; an image processing module configured to perform image processing on each sample image by using an image processing model to obtain a first prediction result and a second prediction result of each sample image, comprising: performing feature extraction on each sample image to obtain sample features corresponding to each sample image; for each sample image, inputting the sample features corresponding to the sample image into an image processing network related to the perception task to which the sample image belongs to obtain a first prediction result corresponding to the sample image; and inputting the sample features corresponding to each sample image into a preset task classifier respectively to obtain a second prediction result corresponding to each sample image; wherein the first prediction result of each sample image is used to represent a prediction result of the sample image with respect to at least one of the perception tasks, and the second prediction result of each sample image is used to represent the perception task to which the sample image belongs. The determining module is configured to determine a target loss based on a first difference between the first prediction result of each sample image and a labeled result in each sample label, and a second difference between the second prediction result of each sample image and a sample identification in each sample label. The adjusting module is configured to adjust parameters in the image processing model by using the target loss to obtain a trained image processing model.

8. An electronic device, comprising: The method comprises: A memory and a processor, wherein the memory stores program instructions, and the processor fetches the program instructions from the memory to execute the method in any one of claims 1-6.

9. A computer-readable storage medium having stored thereon program instructions, wherein, The program instructions are executed by the processor to implement the method in any one of claims 1-6.

Images