Image processing model training method and device, equipment and storage medium

Abstract

The invention discloses an image processing model training method and device, equipment and a storage medium. The training method of the image processing model comprises the following steps: acquiring sample images related to a plurality of perception tasks of a vehicle and sample labels corresponding to the sample images; performing image processing on each sample image by using an image processing model to obtain a first prediction result and a second prediction result of each sample image; determining a target loss based on a first difference between a first prediction result of each sample image and a labeling result in each sample label and a second difference between a second prediction result of each sample image and a sample identifier in each sample label; and adjusting parameters in the image processing model by using the target loss to obtain a trained image processing model. According to the scheme, the prediction performance of the trained image processing model can be improved.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a training method, device, equipment and storage medium for an image processing model. Background Art

[0002] Perception tasks in the field of autonomous driving usually use multiple sensors to perceive various environmental information around the vehicle body, and then process the sensor data through the more mainstream computer vision technology in recent years to obtain information about the understanding of the environment around the vehicle body. Different network structures are usually set for different perception tasks to predict the corresponding results. In order to improve the prediction performance of the model, a more complex network structure is designed, which makes it impossible to meet the system requirements for real-time processing. Therefore, designing the same network model to perceive and predict multiple perception tasks is the current mainstream development of autonomous driving technology. However, for the training of network models that perceive and predict multiple perception tasks at the same time, it is usually adopted to first train the network layers of a single perception task one by one, and then jointly train multiple tasks to obtain a trained network model. In this case, it is easy for different perception tasks to make inconsistent contributions to the gradient of the network shared layer, which leads to the poor prediction performance of the trained network model.

[0003] In view of the existing technical defects, how to provide a model training solution with better prediction performance is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0004] The present application at least provides a training method, apparatus, device and storage medium for an image processing model.

[0005] The present application provides a training method for an image processing model, comprising: obtaining sample images related to several perception tasks of a vehicle and sample labels corresponding to each sample image, wherein the sample label of each sample image includes a sample identifier of the perception task to which the sample image belongs and a labeling result related to the perception task in the sample image; performing image processing on each sample image using an image processing model to obtain a first prediction result and a second prediction result of each sample image, wherein the first prediction result of each sample image is used to represent the prediction result of the sample image with respect to at least one perception task, and the second prediction result of each sample image is used to represent the perception task to which the sample image belongs; determining a target loss based on a first difference between the first prediction result of each sample image and the labeling result in each sample label, and a second difference between the second prediction result of each sample image and the sample identifier in each sample label; and adjusting parameters in the image processing model using the target loss to obtain a trained image processing model.

[0006] In some embodiments, an image processing model is used to perform image processing on each sample image to obtain a first prediction result and a second prediction result for each sample image, including: performing feature extraction on each sample image to obtain sample features corresponding to each sample image; for each sample image, inputting the sample features corresponding to the sample image into an image processing network related to the perception task to which the sample image belongs to obtain a first prediction result corresponding to the sample image; and inputting the sample features corresponding to each sample image into a preset task classifier to obtain a second prediction result corresponding to each sample image.

[0007] In some embodiments, the sample features corresponding to each sample image are respectively input into a preset task classifier to obtain a second prediction result corresponding to each sample image, including: performing the following steps for each sample image: performing convolution processing on the sample features of the sample image to obtain the convolution features of the sample image; performing dimensionality transformation processing on the convolution features of the sample image to obtain transformed features corresponding to the sample image; and determining the second prediction result corresponding to the sample image based on the transformed features corresponding to the sample image.

[0008] In some embodiments, the convolution features of the sample image are dimensionally transformed to obtain transformed features corresponding to the sample image, including: performing a global average pooling operation on the convolution features of the sample image to obtain pooled features corresponding to the sample image; performing a first dimension transformation on the pooled features to obtain first dimension transformed features corresponding to the sample image; performing a second dimension transformation on the first dimension transformed features corresponding to the sample image to obtain second dimension transformed features corresponding to the sample image, and using the second dimension transformed features corresponding to the sample image as the transformed features corresponding to the sample image.

[0009] In some embodiments, the step of performing convolution processing on sample features of a sample image to obtain convolution features of the sample image is executed by a preset convolution module in an image processing model, the preset convolution module includes a plurality of preset convolution layers arranged in cascade, the number of the plurality of preset convolution layers is the same as the number of perception tasks, the features output by the last preset convolution layer are used as the convolution features of the sample image, and the preset convolution module performs convolution processing on the sample features of the sample image input into the preset convolution module to obtain the convolution features of the sample image, including: performing two-dimensional convolution processing on the sample features of the sample image or the features output by the previous preset convolution layer as the input of the current preset convolution layer to obtain two-dimensional convolution features; normalizing the two-dimensional convolution features to obtain normalized features; performing nonlinear processing on the normalized features to obtain processed features, and using the processed features as the features output by the current preset convolution layer.

[0010] In some embodiments, based on a first difference between a first prediction result of each sample image and a labeled result in each sample label, and a second difference between a second prediction result of each sample image and a sample identifier in each sample label, a target loss is determined, including: determining a first loss based on the first difference; determining a second loss based on the second difference; and performing a weighted summation of the first loss and the second loss to obtain a target loss.

[0011] In some embodiments, the training method of the above-mentioned image processing model also includes: acquiring the image to be detected; performing image processing on the image to be detected using the trained image processing model to obtain an image processing result of the image to be detected.

[0012] The present application provides a training device for an image processing model, comprising: an acquisition module, an image processing module, a determination module and an adjustment module; the acquisition module is used to acquire sample images related to several perception tasks of a vehicle and sample labels corresponding to each sample image, the sample label of each sample image includes a sample identifier of the perception task to which the sample image belongs and a labeling result related to the perception task in the sample image; the image processing module is used to perform image processing on each sample image using an image processing model to obtain a first prediction result and a second prediction result of each sample image, the first prediction result of each sample image is used to represent the prediction result of the sample image with respect to at least one perception task, and the second prediction result of each sample image is used to represent the perception task to which the sample image belongs; the determination module is used to determine a target loss based on a first difference between the first prediction result of each sample image and the labeling result in each sample label, and a second difference between the second prediction result of each sample image and the sample identifier in each sample label; the adjustment module is used to adjust parameters in the image processing model using the target loss to obtain a trained image processing model.

[0013] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the training method of the above-mentioned image processing model.

[0014] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the training method of the above-mentioned image processing model is implemented.

[0015] The above scheme uses an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, and uses a first difference between the first prediction result of each sample image regarding at least one perception task and the annotation result in each sample label, and a second difference between the second prediction result obtained for each sample image regarding the perception task to which the sample image belongs and the sample identifier in each sample label to determine the target loss to adjust the parameters in the image processing model to obtain a trained image processing model. Compared with the prior art that only uses the first difference to determine the target loss, the target loss combined with the first difference and the second difference in the present application can be more in line with the loss required for each parameter in the image processing model, and thus the prediction performance of the trained image processing model obtained by using the target loss to adjust is better.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0018] Figure 1 This is a flow chart of an embodiment of a training method for an image processing model of the present application. Figure 1 ;

[0019] Figure 2 This is a flow chart of an embodiment of a training method for an image processing model of the present application. Figure 2 ;

[0020] Figure 3 This is a flow chart of an embodiment of a training method for an image processing model of the present application. Figure 3 ;

[0021] Figure 4 is a schematic diagram of an image processing model in an embodiment of a training method for an image processing model of the present application;

[0022] Figure 5 It is a structural schematic diagram of an embodiment of a training device for an image processing model of the present application;

[0023] Figure 6 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0024] Figure 7 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0025] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0026] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0027] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0028] The present application provides some training methods for image processing models and training devices for image processing models. The application scenarios of the training method for the image processing model include but are not limited to the training process of the detection model related to the vehicle perception task. The executor of the training method for the image processing model may be the training device for the image processing model. For example, the training device for the image processing model may be arranged in a terminal device or a server or other processing device, wherein the terminal device may be a device for training the image processing model, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, etc. In some possible implementations, the training method for the image processing model may be implemented by a processor calling computer-readable instructions stored in a memory.

[0029] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a training method for an image processing model of the present application. Figure 1 Specifically, the training method of the image processing model may include the following steps:

[0030] Step S11: Obtain sample images related to several perception tasks of the vehicle and sample labels corresponding to each sample image.

[0031] The sample label of each sample image includes the sample identifier of the perception task to which the sample image belongs and the annotation results related to the perception task in the sample image.

[0032] Several perception tasks of a vehicle may be tasks related to image detection in traffic scenes related to the vehicle. Among them, several may be one or more. Each perception task may be a preset detection of an image captured by the image acquisition device of the vehicle. Exemplarily, the perception task may be dynamic obstacle detection, static obstacle detection, traffic light recognition, traffic sign recognition, lane line ground marking detection, vehicle scene detection, drivable area segmentation, and detection of whether the vehicle's image acquisition device is blocked, and other task types related to vehicle perception. The present application takes several perception tasks as three perception tasks as an example, and the several perception tasks include perception task A, perception task B, and perception task C. This is only for illustration, and the types and quantities of the several perception tasks are not limited.

[0033] The sample image may be training image data obtained by pre-labeling images collected by vehicles in traffic scenes. Among them, different sample images may be training image data corresponding to different perception tasks. The sample label corresponding to the sample image may include the sample identifier corresponding to the sample image and the labeling result of the sample image. Specifically, the sample label corresponding to the sample image may be used to indicate the sample identifier related to at least one perception task obtained by labeling the sample image and / or the training image data of the sample image belonging to at least one perception task. For example, if a sample image is used to train the image processing model for perception task A, the sample identifier of the sample image may be perception task A, or if a sample image is used to train the image processing model for perception tasks B and C, the sample identifier of the sample image may be perception tasks B and C. Exemplarily, in the case where perception task A is a vehicle scene detection task, the labeling result of the sample image belonging to perception task A may be traffic scene one, traffic scene two, traffic scene three, and so on. Among them, the labeling result corresponding to the sample image may be the labeling of the sample image belonging to the traffic scene, which may specifically be a traffic scene category such as a vehicle in a tunnel scene, a vehicle in an elevated scene, and a vehicle in an urban scene. Wherein, the sample identification of the sample image may be the type of perception task to which the sample image belongs, specifically, whether the sample image is the training image data of perception task A. Exemplarily, in the case where perception task B is a drivable area segmentation task, the annotation result of the sample image belonging to perception task B may indicate whether the image area of ​​the sample image includes or does not include the road surface. Wherein, the annotation result corresponding to the sample image may be an annotation of whether the image area of ​​the sample image belongs to the road surface or does not belong to the road surface, specifically, if at least part of the image area in the image belongs to the road surface, then the image area is a drivable area, otherwise, the image area is a non-drivable area. Wherein, the sample identification of the sample image may be the type of perception task to which the sample image belongs, specifically, whether the sample image is the training image data of perception task B. Exemplarily, in the case where perception task C is a task of detecting whether the image acquisition device of the vehicle is blocked, the annotation result of the sample image belonging to perception task C may be acquired when the image acquisition device of the sample image is blocked, or may be acquired when the image acquisition device of the sample image is not blocked. The annotation result corresponding to the sample image may be an annotation of whether the image area of ​​the sample image is blocked by a specific object, specifically, if at least part of the image area in the image is blocked by a specific object, then the image acquisition device that acquires the sample image is blocked, otherwise, the image acquisition device that acquires the sample image is not blocked. For example, the specific object may be an animal, soil, stain, etc. The sample identifier of the sample image may be the type of perception task to which the sample image belongs, specifically, whether the sample image is training image data for perception task C.

[0034] The method of obtaining each sample image and the sample label corresponding to each sample image can be to obtain training image data related to the vehicle perception task from a public data set, or to directly obtain it from a preset database. Specifically, the image collected by the vehicle is used as a sample image and each sample image is automatically pre-labeled to obtain a sample label and stored in the preset database.

[0035] Step S12: performing image processing on each sample image using the image processing model to obtain a first prediction result and a second prediction result for each sample image.

[0036] The first prediction result of each sample image is used to represent the prediction result of the sample image with respect to at least one perceptual task, and the second prediction result of each sample image is used to represent the perceptual task to which the sample image belongs.

[0037] In some application scenarios, image processing may be performing at least one image processing operation related to a perception task on a sample image. Depending on the perception task, the image processing operation in the image processing model is different. Specifically, the image processing corresponding to the image processing model may be first performing feature extraction on the sample image, and then performing a processing operation matching the perception task on the image after feature extraction. It is understandable that, in the case of different perception tasks, all sample images will be input into the same preset feature extraction network for feature extraction. The preset feature extraction network in the image processing model may be a MobileNet feature extraction network, an Inception feature extraction network, etc. The setting method of the feature extraction network is not limited here. In some application scenarios, all sample images will be input into the same preset feature extraction network for feature extraction, and all sample images after feature extraction will be subjected to processing operations matching all perception tasks to obtain the first prediction results of all sample images. It is understandable that each sample image after feature extraction will perform image processing operations matching all perception tasks. In other application scenarios, all sample images are input into the same preset feature extraction network for feature extraction, and only the processing operation matching the perception task corresponding to the sample image is performed on all sample images after feature extraction to obtain the first prediction result of each sample image. It is understandable that each sample image will only perform the image processing operation matching the perception task corresponding to the sample annotation of the sample image. Exemplarily, for the sample image of perception task A, the processing operation matching perception task A is performed on the sample image after feature extraction to obtain the first prediction result of the sample image. Among them, the processing operation matching perception task A can be the detection of the scene in which the sample image after feature extraction is located.

[0038] In other application scenarios, image processing may be image classification processing of a sample image to obtain a second prediction result of the sample image. Specifically, the image processing corresponding to the image processing model may be first performing feature extraction on the sample image, and then performing classification processing on the image after feature extraction to obtain the second prediction result of the sample image. The second prediction result of the sample image may be training image data indicating that the sample image belongs to at least one of several perception tasks. Exemplarily, for a sample image of perception task A, a classification process is performed on the sample image after feature extraction to obtain the second prediction result of the sample image. Among them, for different sample images, each sample image after feature extraction will perform the same classification process to obtain the second prediction result of each sample image. For a sample image of perception task A, the second prediction result of the sample image is used to indicate whether the sample image belongs to the training image data belonging to perception task A.

[0039] Step S13: Determine the target loss based on a first difference between the first prediction result of each sample image and the annotation result in each sample label, and a second difference between the second prediction result of each sample image and the sample identifier in each sample label.

[0040] The first difference can be used to evaluate the performance of the image processing model on a specific perceptual task. The second difference can be used to evaluate the performance of the image processing model on the classification of perceptual tasks. Specifically, the second difference can be used to evaluate the performance of the shared layer in the image processing model on the classification of perceptual tasks. Exemplarily, the shared layer in the image processing model is used to represent a preset feature extraction network that will be used for the sample image regardless of the type of perceptual task to which the sample image belongs. The target loss can be an overall loss function that combines the first difference and the second difference to guide the optimization of the image processing model.

[0041] Step S14: Use the target loss to adjust the parameters in the image processing model to obtain a trained image processing model.

[0042] The parameters in the image processing model may be parameters of a network or module included in the image processing model. Using the target loss to optimize the image processing model may be to adjust the parameters in the image processing model using a gradient descent method or an adaptive learning rate adjustment, and train the image processing model with the target loss reduced as the goal, until the target loss output by the trained image processing model meets the requirements. Among them, the gradient descent method may be, for example, stochastic gradient descent (SGD), Adam, etc., to adjust the model parameters by back propagation. Among them, the adaptive learning rate adjustment may be to dynamically adjust the target loss according to the training process.

[0043] It can be considered that the target loss combined with the first difference and the second difference used in the present application can better fit the loss required by each parameter in the image processing model, and can additionally train the performance of the shared layer in the image processing model in the classification of perceptual tasks, so that the sample image after feature extraction output by the shared layer in the image processing model can adapt to the image processing operations matched by a variety of perceptual tasks, that is, the sample image after feature extraction output by the shared layer in the trained image processing model is a feature with a smaller degree of distinction between different perceptual tasks, so as to improve the accuracy of the first prediction result obtained by the image processing operation matching the subsequent different perceptual tasks for the sample image after feature extraction. Specifically, the use of the second loss can make the output features of the shared layer in the image processing model as similar as possible between multiple perceptual tasks, so as to ultimately achieve the shared part in the image processing model can generate input features that are beneficial to the image processing network corresponding to the subsequent different perceptual tasks.

[0044] The above scheme uses an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, and uses a first difference between the first prediction result of each sample image regarding at least one perception task and the annotation result in each sample label, and a second difference between the second prediction result obtained for each sample image regarding the perception task to which the sample image belongs and the sample identifier in each sample label to determine the target loss to adjust the parameters in the image processing model to obtain a trained image processing model. Compared with the prior art that only uses the first difference to determine the target loss, the target loss combined with the first difference and the second difference in the present application can be more in line with the loss required for each parameter in the image processing model, and thus the prediction performance of the trained image processing model obtained by using the target loss to adjust is better.

[0045] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a training method for an image processing model of the present application. Figure 2 .

[0046] In some embodiments, the above step S12 may include the following steps: Step S21: extract features from each sample image to obtain sample features corresponding to each sample image. Step S22: for each sample image, input the sample features corresponding to the sample image into an image processing network related to the perception task to which the sample image belongs, and obtain a first prediction result corresponding to the sample image. Step S23: input the sample features corresponding to each sample image into a preset task classifier to obtain a second prediction result corresponding to each sample image.

[0047] The image processing model may include a preset feature extraction network, an image processing network with the same number of types as a number of perception tasks, and a preset task classifier. Different perception tasks match different image processing networks. For each sample image, the sample features corresponding to the sample image are input into the image processing network matching the perception task to which the sample image belongs to perform the image processing operations matched by the above-mentioned perception tasks, and obtain the first prediction result corresponding to the sample image. Exemplarily, the image processing network may be the perception task A network matching the above-mentioned perception task A, specifically the vehicle scene detection network. The sample features of each sample image are input into the perception task A network to perform the image processing operations matched by the perception task A to obtain the first prediction result. The first prediction result may represent the traffic scene to which the sample image belongs and the probability value of the traffic scene to which it belongs. The traffic scene with the largest probability value among the probability values ​​of each traffic scene is determined as the traffic scene to which the sample image belongs. The probability values ​​of each traffic scene may be used to represent the possibility that the sample image belongs to each traffic scene.

[0048] Exemplarily, the image processing network may be a perception task B network matched with the above-mentioned perception task B, specifically a drivable area segmentation network. The sample features of each sample image are input into the perception task B network to perform the image processing operation matched by the perception task B to obtain the first prediction result. Among them, the first prediction result may indicate that at least part of the image area is a drivable area or a non-drivable area. Specifically, the first prediction result includes whether at least part of the image area is a drivable area and the probability value corresponding to each image area. For the sample image, each detection frame obtained through target detection may be an image area contained in the sample image, and the probability value corresponding to each image area is used to indicate the possibility that the image area belongs to the drivable area.

[0049] Exemplarily, the image processing network can be a perception task C network matched with the above-mentioned perception task C, specifically a network for detecting whether the image acquisition device of the vehicle is blocked. The sample features of each sample image are input into the perception task C network to perform the image processing operation matched by the perception task C to obtain the first prediction result. Among them, the first prediction result can indicate whether the image area of ​​the sample image is blocked by a specific object. Specifically, the first prediction result includes the probability value of the sample image being blocked by each specific object, and the probability value of the occlusion of each specific object is used to indicate the possibility of the sample image being blocked by each specific object. Among the probability values ​​of the occlusion of the specific object, the one with the largest probability value is determined as the specific object that blocks the image acquisition device in the first prediction result.

[0050] The above step S23 may be to input the sample features of each sample image into a preset task classifier in batches for classification processing to obtain a second prediction result for each sample image. The preset task classifier may be a preset multi-task classification network. Specifically, the sample features of each sample image are directly input into the fully connected layer to obtain the output of the fully connected layer. The output of the fully connected layer is processed by an activation function to convert the output into a probability distribution to obtain probability values ​​for each category. Each category may be a category to which the sample image belongs to each perception task. The category with the largest probability value among the probability values ​​of each category is determined as the perception task to which the sample image belongs.

[0051] The preset feature extraction network may perform the above step S21. After the above step S21, each image processing network may perform the above step S22. After the above step S21, the preset task classifier may perform the above step S23. The above step S22 and the above step S23 may be performed in parallel.

[0052] It can be considered that each image processing network can focus on its specific perceptual task without being disturbed by other perceptual tasks. For each sample image, the sample features corresponding to the sample image are only input into the image processing network that matches the perceptual task to obtain the first prediction result corresponding to the sample image. The first prediction result is used to determine the target loss to train the image processing model, so that the image processing model can better learn the features related to the specific perceptual task, thereby improving the performance of the trained image processing model in the image processing network corresponding to different perceptual tasks to perform image processing operations matching the perceptual task.

[0053] See also Figure 3 , Figure 3 This is a flow chart of an embodiment of a training method for an image processing model of the present application. Figure 3 .

[0054] In some embodiments, the above step S23 may include the following steps: for each sample image, perform the following Figure 3 The following steps are shown:

[0055] Step S31: performing convolution processing on the sample features of the sample image to obtain the convolution features of the sample image.

[0056] The preset task classifier includes a preset convolution module. The sample features of the sample image are input into the preset convolution module for convolution processing to obtain the convolution features of the sample image. Among them, the preset convolution module may include several preset convolution layers arranged in cascade. Among them, the structure and parameters of each preset convolution layer may be the same or different. Each preset convolution layer may correspond to a convolution processing method. Specifically, the convolution processing method may be any one or at least one of 3D convolution processing, 1×1 convolution processing, transposed convolution processing, and point-by-point grouped convolution. It can be understood that the setting method of several preset convolution layers in the preset convolution module is not limited here.

[0057] In some embodiments, the above step S31 is performed by a preset convolution module in an image processing model, and the preset convolution module includes several preset convolution layers in a cascade arrangement, the number of the preset convolution layers is the same as the number of perception tasks, and the features output by the last preset convolution layer are used as the convolution features of the sample image. The preset convolution module performs the above-mentioned convolution processing on the sample features of the sample image input into the preset convolution module to obtain the convolution features of the sample image, including: using the sample features of the sample image or the features output by the previous preset convolution layer as the input of the current preset convolution layer for two-dimensional convolution processing to obtain two-dimensional convolution features. The two-dimensional convolution features are normalized to obtain normalized features. The normalized features are nonlinearly processed to obtain processed features, and the processed features are used as the features output by the current preset convolution layer.

[0058] Among them, the structures of the preset convolutional layers in the cascaded setting are the same, but the parameters can be the same or different. The current preset convolutional layer can be any preset convolutional layer in the preset convolution module. Exemplarily, the current preset convolutional layer can be the first preset convolutional layer or a non-first preset convolutional layer.

[0059] Exemplarily, when the current preset convolution layer is the first preset convolution layer, the sample features of the sample image are used as the input of the current preset convolution layer for two-dimensional convolution processing to obtain two-dimensional convolution features. The sample features of the sample image are input into the first preset convolution layer for two-dimensional convolution processing to obtain the two-dimensional convolution features of the first preset convolution layer. The two-dimensional convolution features of the first preset convolution layer are normalized to obtain the normalized features of the first preset convolution layer. For example, the normalization process can be represented as BN processing. The normalized features are nonlinearly processed to obtain processed features, and the processed features are used as the features output by the first preset convolution layer. Among them, the nonlinear processing can be to use a preset activation function to perform a nonlinear transformation on the normalized features of the first preset convolution layer to obtain the processed features. Among them, the preset activation function can be set to a ReLU activation function, a Sigmoid activation function, and a Tanh activation function, etc.

[0060] Exemplarily, when the current preset convolution layer is not the first preset convolution layer, for example, when the current preset convolution layer is the last preset convolution layer, the sample features of the sample image are used as the input of the current preset convolution layer for two-dimensional convolution processing to obtain two-dimensional convolution features. The sample features of the sample image are input into the last preset convolution layer for two-dimensional convolution processing to obtain the two-dimensional convolution features of the last preset convolution layer. The two-dimensional convolution features of the last preset convolution layer are normalized to obtain the normalized features of the last preset convolution layer. The normalized features are nonlinearly processed to obtain processed features, and the processed features are used as the features output by the last preset convolution layer. The nonlinear processing can be to use a preset activation function to perform a nonlinear transformation on the normalized features of the last preset convolution layer to obtain the processed features.

[0061] It can be considered that, according to the number of types of perception tasks, setting the preset convolution layer in the preset convolution module can enable the image processing model to enhance the important features in the sample image for each sample feature, thereby improving the accuracy of image data for determining whether the sample image belongs to any perception task.

[0062] Step S32: Perform dimensionality transformation on the convolution features of the sample image to obtain transformed features corresponding to the sample image.

[0063] The preset task classifier includes a preset dimension transformation module, and the preset dimension transformation module includes a preset dimension transformation layer in a cascade arrangement.

[0064] The convolution features of the sample image are input into the preset dimension transformation module for dimension transformation processing to obtain the transformed features corresponding to the sample image. Among them, the structure and parameters of each preset dimension transformation layer can be the same or different. Each preset dimension transformation layer can correspond to a dimension transformation method. Exemplarily, the preset dimension transformation layer can be a linear layer. The preset dimension transformation module is used to convert the convolution features to the target dimension in order to perform the classification task. Specifically, the preset dimension transformation layer can be a fully connected layer (Fully Connected Layer), a linear layer (Linear Layer) and a multilayer perceptron MLP layer (Multilayer Perceptron, MLP). It can be understood that the setting method of several preset dimension transformation layers in the preset dimension transformation module is not limited here.

[0065] In some embodiments, the above step S32 may include the following steps: first, perform a global average pooling operation on the convolution features of the sample image to obtain the pooled features corresponding to the sample image. Then, perform a first dimensional transformation on the pooled features to obtain the first dimensional transformed features corresponding to the sample image. Next, perform a second dimensional transformation on the first dimensional transformed features corresponding to the sample image to obtain the second dimensional transformed features corresponding to the sample image, and use the second dimensional transformed features corresponding to the sample image as the transformed features corresponding to the sample image.

[0066] Exemplarily, the input of the task classifier is a preset feature extraction network in a shared layer that extracts features from a sample image to obtain sample features. Among them, the dimensions of the sample features are (N, C, H, W), where N is the mini-batch size, C is the number of feature channels, and H and W are height and width, respectively. When the sample features pass through the preset convolution module, the dimensions of the convolution features obtained do not change. The preset dimension transformation module includes a first linear layer and a second linear layer. Both the first linear layer and the second linear layer can be represented as linear layers. For the first linear layer, its input channel number is C and the output channel number is 2048. For the second linear layer, its input and output channel number is 2048, and its output channel number is the number of types of perception tasks. If the number of types of perception tasks is 4, the number of output channels is 4. If the number of types of perception tasks is 3, the number of output channels is 3. The preset dimension transformation module may also include a pooling layer and a feature transformation layer. Among them, the pooling layer may be used to perform global average pooling processing on the features of the input pooling layer. The feature transformation layer may be an output of the feature transformation layer obtained by performing feature transformation on the input features using a view function. The pooled features are input into the feature transformation layer for first dimension transformation processing to obtain the features after first dimension transformation corresponding to the sample image. The features after first dimension transformation corresponding to the sample image are input into the first linear layer and the second linear layer for second dimension transformation processing to obtain the features after second dimension transformation corresponding to the sample image.

[0067] Step S33: Determine a second prediction result corresponding to the sample image based on the transformed features corresponding to the sample image.

[0068] The transformed features corresponding to each sample image are processed by an activation function to convert the output into a probability distribution to obtain probability values ​​of each category. Each category can be a category to which the sample image belongs to each perception task. The category with the largest probability value among the probability values ​​of each category is determined as the perception task to which the sample image belongs. Specifically, the category with the largest probability value among the probability values ​​of each category is determined as the second prediction result of the sample image. Among them, the preset activation function can be set to a ReLU activation function, a Sigmoid activation function, a Tanh activation function, etc.

[0069] It can be considered that by classifying each sample feature into its corresponding perceptual task using a preset task classifier, the accuracy of the second prediction result can be improved, thereby making the parameters of the shared layer in the image processing model trained using the target loss more in line with the training requirements, thereby improving the prediction performance of the trained image processing model.

[0070] In some embodiments, the above step S13 may include the following steps: determining a first loss based on the first difference, determining a second loss based on the second difference, and performing a weighted summation of the first loss and the second loss to obtain a target loss.

[0071] Specifically, the method of calculating the target loss can refer to formula (1): TL = w1×loss1+…+wn×lossn+w0×(L1+…+Ln) formula (1);

[0072] Among them, TL can be used to represent the target loss. lossn can represent the first difference obtained from the sample image corresponding to the nth perception task, and the nth first loss determined. wn can be used to represent the weight of n first losses. loss1 can represent the first difference obtained from the sample image corresponding to any perception task, and any first loss determined. w1 can be used to represent the weight of any first loss. Ln can represent the second difference obtained from the sample image corresponding to the nth perception task, and the nth second loss determined. L1 can represent the second difference obtained from the sample image corresponding to any perception task, and any second loss determined. w0 can represent the weights corresponding to all second losses.

[0073] Exemplarily, for each sample image, the sample identifier of the perception task to which the sample image belongs adopts the One-hot encoding form. For example, if the sample image is training image data belonging to perception task A, the sample identifier of the sample image is set to the first encoding result (1, 0, 0). If the sample image is training image data belonging to perception task B, the sample identifier of the sample image is set to the second encoding result (0, 1, 0). If the sample image is training image data belonging to perception task C, the sample identifier of the sample image is set to the third encoding result (0, 0, 1). The present application can set the mini-batch to 1, that is, the output dimension of the task classifier is (1, 3). It is assumed that the sample image belonging to perception task A is subjected to feature extraction through a preset feature extraction network, and then the second prediction result output by the preset task classifier is (0.8, 0.1, 0.2). The second loss of the sample image can be obtained by calculating the loss function between the second prediction result (0.8, 0.1, 0.2) and the first encoding result (1, 0, 0). It can be understood that the second loss of sample images belonging to other perception tasks is analogous.

[0074] In some application scenarios, a cross entropy loss function is used to define a second loss corresponding to a preset task classifier. Specifically, the process of determining the second loss based on the second difference can refer to formula (2):

[0075]

[0076] Where C is the number of task categories, which can be the number of types of several perception tasks, that is, the number of perception tasks. c and x i can represent the output results of the above preset task classifier when the task category is i and the task category is c. Similarly, y c It can be expressed as the task identifier corresponding to the task category c. For example, y c It can be used to represent the sample identifier of any sample image when the task category is c. loss can be used to represent the second loss of any sample image. c It can be used to represent the sample identifier of any sample image. The sample identifier is usually obtained using the one-hot encoding form mentioned above.

[0077] In some embodiments, the above-mentioned image processing model training method further includes: obtaining an image to be detected, performing image processing on the image to be detected using the trained image processing model, and obtaining an image processing result of the image to be detected.

[0078] The image to be detected may be an image that needs to perform image processing related to the above-mentioned several perception tasks. The image to be detected may be an image acquired by an image acquisition device of a vehicle. The image processing result of the image to be detected may be a prediction result of the perception task to which the image to be detected belongs obtained by performing image processing related to the perception task on the image to be detected.

[0079] The image to be detected is input into the trained image processing model for image processing to obtain the image processing result of the image to be detected. Specifically, the image to be detected is input into the preset feature extraction network in the trained image processing model for feature extraction to obtain the image to be detected after feature extraction. The image to be detected after feature extraction is input into the image processing network that matches the perception task in the image processing model for image processing to obtain the prediction result of the image to be detected. It can be understood that the above-mentioned input of the image to be detected into the preset feature extraction network in the trained image processing model for feature extraction to obtain the image to be detected after feature extraction can refer to the above-mentioned step S21, which will not be repeated here. The above-mentioned input of the image to be detected after feature extraction into the image processing network that matches the perception task in the image processing model for image processing to obtain the prediction result of the image to be detected can refer to the above-mentioned step S22, which will not be repeated here.

[0080] In some application scenarios, an image to be detected and a perception task matching the image to be detected are obtained. The perception task matching the image to be detected can be obtained by parsing the operating instructions on the display interface of the vehicle to obtain the perception task matching the image to be detected. The image to be detected is processed using a trained image processing model to obtain an image processing result of the image to be detected. The image to be detected after feature extraction is input into the image processing network in the image processing model that matches the perception task matching the image to be detected for image processing to obtain a prediction result of the image to be detected.

[0081] See also Figure 4 , Figure 4 It is a schematic diagram of an image processing model in an embodiment of a training method for an image processing model of the present application.

[0082] The image processing model includes a shared layer, an image processing network matching each perception task, and a preset task classifier. Among them, the shared layer can be the above-mentioned preset feature extraction network. The image processing networks matching each perception task can be the above-mentioned perception task A network, the above-mentioned perception task B network, and the above-mentioned perception task C network. The preset task classifier includes a preset convolution module and a preset dimensional transformation module in a cascade setting. Specifically, please refer to the above content for the setting of the network and module structure and the functions performed in the preset convolution module and the preset dimensional transformation module, which will not be repeated here. It can be understood that in the image processing model, the image processing networks matching each perception task are at least two image processing networks. This is only for example, and does not limit the number of each perception task in the image processing model.

[0083] Exemplarily, sample image A, sample image B and sample image C may be training image data whose samples are identified as belonging to perception task A, perception task B and perception task C, respectively. Among them, sample image A, sample image B and sample image C may include one sample image or multiple sample images, respectively. For example, taking sample image A as a sample image, sample image A is input into the shared layer, and step S21 is executed to obtain the sample features of sample image A output by the shared layer. In some application scenarios, the sample features of sample image A are input into the above-mentioned perception task A network, and step S22 is executed to obtain the first prediction result corresponding to sample image A. Then, based on the first difference between the first prediction result of each sample image and the annotation result in each sample label, the first loss is determined. In other application scenarios, the sample features of sample image A are input into the above-mentioned preset task classifier, and step S23 is executed to obtain the second prediction result corresponding to sample image A. Specifically, the preset dimension transformation module in the above-mentioned preset task classifier includes a pooling layer, a feature transformation layer, a first linear layer and a second linear layer. Among them, the pooling layer, feature transformation layer, first linear layer and second linear layer can all refer to the above content, which will not be repeated here. The convolution features output by the preset convolution module are sequentially input into the pooling layer, feature transformation layer, first linear layer and second linear layer to obtain the features after the second dimension transformation output by the second linear layer. And the features after the second dimension transformation are used as the transformed features corresponding to the sample image A. Then, based on the second difference between the second prediction results of each sample image and the sample identification in each sample label, the second loss is determined. The processing process of sample image B and sample image C can refer to the above content, which will not be repeated here.

[0084] The first loss of all sample images and the second loss of all sample images are weightedly summed to obtain a target loss. Using the target loss, at least some modules or network layers in the image processing model are adjusted by back propagation. For example, only the parameters of the image processing model including the shared layer, the image processing network matching each perception task, and the preset part in the preset task classifier are adjusted. The preset part in the preset task classifier can be all layers other than the pooling layer in the preset convolution module and the preset dimensional transformation module.

[0085] The above scheme uses an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, and uses a first difference between the first prediction result of each sample image regarding at least one perception task and the annotation result in each sample label, and a second difference between the second prediction result obtained for each sample image regarding the perception task to which the sample image belongs and the sample identifier in each sample label to determine the target loss to adjust the parameters in the image processing model to obtain a trained image processing model. Compared with the prior art that only uses the first difference to determine the target loss, the target loss combined with the first difference and the second difference in the present application can be more in line with the loss required for each parameter in the image processing model, and thus the prediction performance of the trained image processing model obtained by using the target loss to adjust is better.

[0086] See also Figure 5 , Figure 5 It is a structural schematic diagram of an embodiment of a training device for an image processing model of the present application. The training device 50 for an image processing model includes an acquisition module 51, an image processing module 52, a determination module 53, and an adjustment module 54. The acquisition module 51 is used to acquire sample images related to several perception tasks of a vehicle and sample labels corresponding to each sample image. The sample label of each sample image includes a sample identifier of the perception task to which the sample image belongs and a labeling result related to the perception task in the sample image; the image processing module 52 is used to perform image processing on each sample image using an image processing model to obtain a first prediction result and a second prediction result of each sample image. The first prediction result of each sample image is used to represent the prediction result of the sample image with respect to at least one perception task, and the second prediction result of each sample image is used to represent the perception task to which the sample image belongs; the determination module 53 is used to determine a target loss based on a first difference between the first prediction result of each sample image and the labeling result in each sample label, and a second difference between the second prediction result of each sample image and the sample identifier in each sample label; the adjustment module 54 is used to adjust the parameters in the image processing model using the target loss to obtain a trained image processing model.

[0087] The above scheme uses an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, and uses a first difference between the first prediction result of each sample image regarding at least one perception task and the annotation result in each sample label, and a second difference between the second prediction result obtained for each sample image regarding the perception task to which the sample image belongs and the sample identifier in each sample label to determine the target loss to adjust the parameters in the image processing model to obtain a trained image processing model. Compared with the prior art that only uses the first difference to determine the target loss, the target loss combined with the first difference and the second difference in the present application can be more in line with the loss required for each parameter in the image processing model, and thus the prediction performance of the trained image processing model obtained by using the target loss to adjust is better.

[0088] For the functions performed by each module, please refer to the training method of the image processing model, which will not be repeated here.

[0089] See also Figure 6 , Figure 6 6 is a schematic diagram of the structure of an embodiment of a vehicle of the present application. The electronic device 60 includes a memory 61 and a processor 62, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps in the above-mentioned training method embodiment of the image processing model. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 60 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0090] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in the training method embodiment of the above-mentioned image processing model. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0091] The above scheme uses an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, and uses a first difference between the first prediction result of each sample image regarding at least one perception task and the annotation result in each sample label, and a second difference between the second prediction result obtained for each sample image regarding the perception task to which the sample image belongs and the sample identifier in each sample label to determine the target loss to adjust the parameters in the image processing model to obtain a trained image processing model. Compared with the prior art that only uses the first difference to determine the target loss, the target loss combined with the first difference and the second difference in the present application can be more in line with the loss required for each parameter in the image processing model, and thus the prediction performance of the trained image processing model obtained by using the target loss to adjust is better.

[0092] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 701, which, when executed by a processor, implement the steps in any of the above-mentioned training method embodiments for image processing models.

[0093] The above scheme uses an image processing model to perform image processing on each sample image to obtain a first prediction result and a second prediction result of each sample image, and uses a first difference between the first prediction result of each sample image regarding at least one perception task and the annotation result in each sample label, and a second difference between the second prediction result obtained for each sample image regarding the perception task to which the sample image belongs and the sample identifier in each sample label to determine the target loss to adjust the parameters in the image processing model to obtain a trained image processing model. Compared with the prior art that only uses the first difference to determine the target loss, the target loss combined with the first difference and the second difference in the present application can be more in line with the loss required for each parameter in the image processing model, and thus the prediction performance of the trained image processing model obtained by using the target loss to adjust is better.

[0094] In some embodiments, the functions or modules included in the system provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0095] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0096] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0097] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0098] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A training method for an image processing model, characterized in that: The method comprises: Acquire sample images related to several perception tasks of the vehicle and sample labels corresponding to each of the sample images, wherein the sample label of each of the sample images includes a sample identifier of the perception task to which the sample image belongs and a labeling result related to the perception task in the sample image; Performing image processing on each of the sample images using an image processing model to obtain a first prediction result and a second prediction result of each of the sample images, wherein the first prediction result of each of the sample images is used to represent a prediction result of the sample image with respect to at least one of the perceptual tasks, and the second prediction result of each of the sample images is used to represent the perceptual task to which the sample image belongs; Determine a target loss based on a first difference between a first prediction result of each of the sample images and a labeling result in each of the sample tags, and a second difference between a second prediction result of each of the sample images and a sample identifier in each of the sample tags; The target loss is used to adjust the parameters in the image processing model to obtain a trained image processing model.

2. The method according to claim 1, characterized in that The using the image processing model to perform image processing on each of the sample images to obtain the first prediction result and the second prediction result of each of the sample images includes: Extracting features from each of the sample images to obtain sample features corresponding to each of the sample images; For each of the sample images, inputting a sample feature corresponding to the sample image into an image processing network related to the perception task to which the sample image belongs, to obtain a first prediction result corresponding to the sample image; The sample features corresponding to each of the sample images are respectively input into a preset task classifier to obtain a second prediction result corresponding to each of the sample images.

3. The method according to claim 2, characterized in that The step of inputting the sample features corresponding to each of the sample images into a preset task classifier to obtain the second prediction results corresponding to each of the sample images comprises: For each sample image, perform the following steps: Performing convolution processing on the sample features of the sample image to obtain convolution features of the sample image; Performing dimensional transformation processing on the convolution features of the sample image to obtain transformed features corresponding to the sample image; Based on the transformed features corresponding to the sample image, a second prediction result corresponding to the sample image is determined.

4. The method according to claim 3, characterized in that The step of performing dimensional transformation on the convolution features of the sample image to obtain transformed features corresponding to the sample image includes: Performing a global average pooling operation on the convolutional features of the sample image to obtain pooled features corresponding to the sample image; Performing a first-dimensional transformation process on the pooled features to obtain the first-dimensional transformed features corresponding to the sample image; Perform a second-dimensional transformation on the features corresponding to the sample image after the first-dimensional transformation to obtain the features corresponding to the sample image after the second-dimensional transformation, and use the features corresponding to the sample image after the second-dimensional transformation as the transformed features corresponding to the sample image.

5. The method according to claim 3, characterized in that: The step of performing convolution processing on the sample features of the sample image to obtain the convolution features of the sample image is performed by a preset convolution module in the image processing model, the preset convolution module includes a plurality of preset convolution layers arranged in cascade, the number of the plurality of preset convolution layers is the same as the number of perception tasks, the features output by the last preset convolution layer are used as the convolution features of the sample image, and the preset convolution module performs convolution processing on the sample features of the sample image input into the preset convolution module to obtain the convolution features of the sample image, including: Performing two-dimensional convolution processing on the sample features of the sample image or the features output by the previous preset convolution layer as the input of the current preset convolution layer to obtain two-dimensional convolution features; Normalizing the two-dimensional convolutional features to obtain normalized features; The normalized features are subjected to nonlinear processing to obtain processed features, and the processed features are used as features output by the current preset convolutional layer.

6. The method according to any one of claims 1 to 5, characterized in that The determining of the target loss based on a first difference between the first prediction result of each of the sample images and the annotation result in each of the sample tags, and a second difference between the second prediction result of each of the sample images and the sample identifier in each of the sample tags, comprises: determining a first loss based on the first difference; determining a second loss based on the second difference; The first loss and the second loss are weightedly summed to obtain the target loss.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Acquire the image to be detected; The trained image processing model is used to perform image processing on the image to be detected to obtain an image processing result of the image to be detected.

8. A training device for an image processing model, characterized in that: include: An acquisition module, used to acquire sample images related to several perception tasks of the vehicle and sample labels corresponding to each of the sample images, wherein the sample label of each of the sample images includes a sample identifier of the perception task to which the sample image belongs and a labeling result related to the perception task in the sample image; An image processing module, used to perform image processing on each of the sample images using an image processing model to obtain a first prediction result and a second prediction result of each of the sample images, wherein the first prediction result of each of the sample images is used to represent a prediction result of the sample image with respect to at least one of the perceptual tasks, and the second prediction result of each of the sample images is used to represent the perceptual task to which the sample image belongs; A determination module, configured to determine a target loss based on a first difference between a first prediction result of each of the sample images and a labeling result in each of the sample tags, and a second difference between a second prediction result of each of the sample images and a sample identifier in each of the sample tags; An adjustment module is used to adjust the parameters in the image processing model using the target loss to obtain a trained image processing model.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 7.

Images