Image processing method and device, model training method and device and terminal equipment

Abstract

The image processing method comprises the following steps: acquiring a first image; performing feature extraction on the first image through an encoder in the neural network model to obtain n first feature maps, n being greater than or equal to 2, and different first feature maps having different scales; the n first feature maps are fused through a first decoder in a neural network model to obtain a first prediction label, a second decoder is further configured in the neural network model, and the second decoder is used for optimizing the first decoder when the neural network model is trained to obtain a second prediction label; the scale of the feature pattern processed by the second decoder is larger than that of the feature pattern processed by the first decoder; and outputting the first prediction label. Therefore, through the decoupling design of the first decoder and the second decoder, the light weight of the model is completed while the image processing effect is ensured in the image processing process. Therefore, the image processing effect and efficiency of the end side are improved, and the time delay and memory consumption of the end side can be reduced.

Description

Image processing method, model training method, device and terminal equipment Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to an image processing method, a model training method, an apparatus, and a terminal device. Background Art

[0002] Semantic segmentation is one of the basic tasks in computer vision and is also the basic model in autonomous driving perception tasks. It can be used as input for a variety of downstream tasks, such as crowdsourced mapping, crowdsourced positioning, automatic parking, floor positioning, etc.

[0003] Currently, semantic segmentation requires pixel-by-pixel classification of input images, resulting in a high computational load. To reduce the computational load required for semantic segmentation, there are generally two approaches: reducing image size and reducing model complexity. Reducing image size is the most direct way to reduce computational load, but this results in a significant loss of image detail, impacting accuracy. Reducing model complexity weakens the model's feature extraction capabilities, thus affecting segmentation accuracy. Therefore, applying lightweight models to semantic segmentation tasks while balancing real-time performance and accuracy is a pressing technical challenge.

[0004] Summary of the Invention

[0005] The present application provides an image processing method, a model training method, an image processing device, a model training device, a terminal device, a server, a computer storage medium and a computer product, which can realize the application of lightweight models in semantic segmentation tasks while taking into account real-time and accuracy performance.

[0006] In a first aspect, the present application provides an image processing method, the method comprising: acquiring a first image; performing feature extraction on the first image through an encoder in a neural network model to obtain n first feature maps, n≥2, wherein different first feature maps have different scales (shapes); fusing the n first feature maps through a first decoder in the neural network model to obtain a first prediction label, wherein a second decoder is further configured in the neural network model, and the second decoder is used to optimize the first decoder when training the neural network model, and the scale of the feature map processed by the second decoder is larger than the scale of the feature map processed by the first decoder; and outputting the first prediction label.

[0007] In this way, since only the first decoder in the neural network model is used for decoding during image processing, and the scale of the image processed by the first decoder is lower than the scale of the image processed by the second decoder, the computational complexity of the first decoder is smaller than that of the second decoder. At the same time, since the second decoder is used to optimize the first decoder when training the neural network model, the first decoder can achieve better decoding results with a smaller computational complexity. In this way, by decoupling the first and second decoders, the image processing process can achieve lightweight neural network models while ensuring image processing effects. This can improve the image processing effect and efficiency on the end side, thereby reducing end-side latency and memory consumption.

[0008] For example, the scale (shape) can refer to: the calculation result of a certain layer in the neural network, which is a tensor (tensor) during the calculation process. The scale (shape) of the tensor indicates the arrangement (dimension) of the data in the memory and the length of the data in each dimension, such as: 1) After a general 3-channel image is converted into a tensor, its scale is [H, W, C], where H is the image height dimension, W is the width dimension, and C is the number of channels (3); 2) After the image is calculated by the neural network, the tensor expression of the Nth feature map is output, and its scale (shape) is transformed into [H / 16, W / 16, C1].

[0009] For example, executing an event through a module in a neural network model can be understood as executing the code associated with the module in the neural network model. In this case, the module in the neural network model can be understood as, but not limited to, a string of code. For example, extracting features from a first image through an encoder in a neural network model can be understood as extracting features from the first image by executing the code associated with the encoder in the neural network model.

[0010] In some possible implementations, the method can be performed by a vehicle. In this case, the first image is an image captured by the vehicle while driving. The first predicted label includes one or more of the following: lane markings, guide arrows, road traffic markings, road traffic signs, crosswalks, guide strips, and poles on the road.

[0011] In some possible implementations, the method may further include: acquiring a second image; performing data augmentation on the second image to obtain a third image; processing the second predicted label obtained by processing the second image using the neural network model and the third predicted label obtained by processing the third image using the neural network model using a first loss function to obtain a first loss; adding a perturbation to the second image along the gradient ascent direction based on the first loss to obtain a fourth image; processing the second predicted label and the fourth predicted label obtained by processing the fourth image using the neural network model using the first loss function to obtain a second loss; and updating the neural network model based on the second loss. Thus, self-supervised training of the neural network model is achieved in the absence of labels, thereby completing the update of the neural network model. Exemplarily, the first image and the second image may be, but are not limited to, the same image. Exemplarily, when performing data augmentation on the second image, a perturbation may be randomly added to the second image to obtain the third image.

[0012] In some possible implementations, the method may further include: obtaining a fifth image; binarizing the fifth image to obtain a first true value label, the first true value label being used to characterize an edge in the fifth image; processing the first true value label and at least one feature map extracted from the fifth image by the neural network model based on a second loss function to obtain a third loss; and updating the neural network model based on the third loss. Thus, the neural network model is updated using some detail information in the image. Since some detail information of the image can be obtained after binarizing the image, and a loss can be obtained using this detail information, and the model is updated using this loss. Therefore, the updated model can pay more attention to the detail information in the image. This improves the image processing effect of the model. Exemplarily, the edge of the fifth image can be, but is not limited to, obtained from a location in the fifth image where the gradient change is greater than a preset value.

[0013] In a second aspect, the present application provides a model training method, which may include: obtaining a first image; performing feature extraction on the first image through an encoder in a neural network model to obtain n first feature maps, n≥2, wherein different first feature maps have different scales; performing upsampling processing on k first feature maps in the n first feature maps through a first decoder in the neural network model to obtain k second feature maps, k≥1, wherein the scale of each feature map in the k first feature maps is larger than the scale of the remaining (nk) first feature maps in the n first feature maps; fusing the k second feature maps and the (nk) first feature maps through the first decoder to obtain a first prediction label; fusing the k first feature maps and the feature map fused by the first decoder through the second decoder in the neural network model to obtain a second prediction label; obtaining the loss corresponding to the first image according to the first prediction label, the second prediction label and the sample label of the first image; and training the neural network model based on the loss corresponding to the first image.

[0014] In this way, the training of the neural network model is completed. Since the scale of the image processed by the first decoder in the neural network model is lower than the scale of the image processed by the second decoder during the model training process, the computational complexity of the first decoder is smaller than that of the second decoder. At the same time, the second decoder is used to optimize the first decoder, so the first decoder can achieve better decoding results with a smaller computational complexity. In this way, by decoupling the first decoder and the second decoder, the neural network model can be trained to perform image processing using the first decoder with a smaller computational complexity, thereby achieving lightweighting of the neural network model while ensuring the image processing effect during the image processing process. As a result, a lightweight model can be deployed on the end side, thereby improving the image processing effect and efficiency on the end side, and further reducing the latency and memory consumption on the end side.

[0015] In some possible implementations, the loss corresponding to the first image is obtained based on the first predicted label, the second predicted label and the sample label of the first image, specifically including: based on the first loss function, processing the first predicted label and the sample label of the first image to obtain the first loss; based on the second loss function, processing the second predicted label and the sample label of the first image to obtain the second loss; based on the first loss and the second loss, obtaining the loss corresponding to the first image.

[0016] In some possible implementations, the method may further include: binarizing the first image to obtain a first true value label, the first true value label being used to characterize the edge in the first image; processing the first true value label and at least one feature map of the n first feature maps based on a third loss function to obtain a third loss; and training the neural network model based on the third loss. Thus, by utilizing some detail information in the image, auxiliary training of the neural network model is completed. Since some detail information of the image can be obtained after binarizing the image, and a loss can be obtained using this detail information, and the model is trained using this loss. Therefore, the trained model can pay more attention to the detail information in the image. This improves the image processing effect of the model.

[0017] In some possible implementations, the method may further include: obtaining a first gradient associated with the loss corresponding to the first image based on the loss corresponding to the first image; adding a perturbation to the first image based on the first gradient and in the direction of gradient ascent to obtain a second image; and training a neural network model using the sample labels of the second image and the first image. Thus, by using gradient attack, the data at the model's decision boundary is enhanced, and the enhanced data is used for auxiliary training, thereby preventing the trained model from focusing too much on irrelevant or local information in the image, thereby improving the model's performance, such as generalization and robustness.

[0018] In some possible implementations, the method may be performed by a cloud server. In this case, after completing the training of the neural network model, the method may further include: performing semantic segmentation using the encoder and the first decoder, or performing semantic segmentation using the encoder and the second decoder.

[0019] In some possible implementations, the method may be performed by a terminal device. In this case, after completing the training of the neural network model, the method may further include: performing semantic segmentation using an encoder and a first decoder. Exemplarily, the terminal device may be a vehicle.

[0020] In some possible implementations, the method can be performed by a cloud server. In this case, after completing the training of the neural network model, the method can further include: deploying the encoder and the first decoder on a terminal device, so that the terminal device uses the encoder and the first decoder to perform semantic segmentation. For example, the terminal device can be a vehicle.

[0021] In a third aspect, the present application provides an image processing device, which includes: an acquisition module for acquiring a first image; a processing module for performing feature extraction on the first image through an encoder in a neural network model to obtain n first feature maps, n≥2, wherein different first feature maps have different scales; the processing module is also used to fuse the n first feature maps through a first decoder in the neural network model to obtain a first prediction label, wherein the neural network model is also configured with a second decoder, and the second decoder is used to optimize the first decoder when training the neural network model, and the scale of the feature map processed by the second decoder is larger than the scale of the feature map processed by the first decoder; the processing module is also used to output the first prediction label.

[0022] In some possible implementations, the image processing device is deployed in a vehicle, the first image is an image collected by the vehicle while it is driving, and the first predicted label includes one or more of the following: lane lines, guide arrows, road traffic markings, road traffic signs, crosswalk lines, guide strips, or poles on the road.

[0023] In some possible implementations, the acquisition module is also used to acquire a second image; the processing module is also used to perform data enhancement on the second image to obtain a third image; the processing module is also used to process the second predicted label obtained by processing the second image through the neural network model and the third predicted label obtained by processing the third image through the neural network model through the first loss function to obtain a first loss; the processing module is also used to add perturbation to the second image along the gradient ascent direction based on the first loss to obtain a fourth image; the processing module is also used to process the second predicted label and the fourth predicted label obtained by processing the fourth image through the neural network model through the first loss function to obtain a second loss; the processing module is also used to update the neural network model based on the second loss.

[0024] In some possible implementations, the acquisition module is further used to acquire a fifth image; the processing module is further used to binarize the fifth image to obtain a first true value label, and the first true value label is used to characterize the edge in the fifth image; the processing module is further used to process the first true value label and at least one feature map extracted from the fifth image through the neural network model based on a second loss function to obtain a third loss; the processing module is further used to update the neural network model based on the third loss.

[0025] In a fourth aspect, the present application provides a model training device, which includes: an acquisition module for acquiring a first image; a processing module for performing feature extraction on the first image through an encoder in a neural network model to obtain n first feature maps, n≥2, wherein different first feature maps have different scales; the processing module is also used to upsample k first feature maps of the n first feature maps through a first decoder in the neural network model to obtain k second feature maps, k≥1, wherein the scale of each feature map in the k first feature maps is larger than the scale of the remaining (nk) first feature maps in the n first feature maps; the processing module is also used to fuse the k second feature maps and the (nk) first feature maps through the first decoder to obtain a first prediction label; the processing module is also used to fuse the k first feature maps and the feature map fused by the first decoder through the second decoder in the neural network model to obtain a second prediction label; the processing module is also used to obtain the loss corresponding to the first image based on the first prediction label, the second prediction label and the sample label of the first image; the processing module is also used to train the neural network model based on the loss corresponding to the first image.

[0026] In some possible implementations, when the processing module obtains the loss corresponding to the first image based on the first prediction label, the second prediction label and the sample label of the first image, it is specifically used to: based on the first loss function, process the first prediction label and the sample label of the first image to obtain the first loss; based on the second loss function, process the second prediction label and the sample label of the first image to obtain the second loss; based on the first loss and the second loss, obtain the loss corresponding to the first image.

[0027] In some possible implementations, the processing module is further used to: binarize the first image to obtain a first true value label, where the first true value label is used to characterize the edge in the first image; based on a third loss function, process the first true value label and at least one feature map of the n first feature maps to obtain a third loss; and train the neural network model based on the third loss.

[0028] In some possible implementations, the processing module is further used to: obtain a first gradient associated with the loss corresponding to the first image based on the loss corresponding to the first image; add perturbation to the first image based on the first gradient and along the gradient ascent direction to obtain a second image; and train the neural network model using sample labels of the second image and the first image.

[0029] In some possible implementations, the model training device is deployed in a cloud server, and after completing the training of the neural network model, the processing module is also used to: perform semantic segmentation using the encoder and the first decoder, or perform semantic segmentation using the encoder and the second decoder.

[0030] In some possible implementations, the model training device is deployed in a terminal device, and after completing the training of the neural network model, the processing module is further configured to: perform semantic segmentation using the encoder and the first decoder. Exemplarily, the terminal device is a vehicle.

[0031] In some possible implementations, the model training device is deployed in a cloud server, and after completing the training of the neural network model, the processing module is further configured to: deploy the encoder and the first decoder on a terminal device, so that the terminal device uses the encoder and the first decoder to perform semantic segmentation. Exemplarily, the terminal device is a vehicle.

[0032] In a fifth aspect, the present application provides a model training method, comprising: acquiring a first image; performing data enhancement on the first image to obtain a second image; processing the first predicted label obtained by processing the first image by the neural network model and the second predicted label obtained by processing the second image by the neural network model through a first loss function to obtain a first loss; based on the first loss, adding perturbations to the first image along the gradient ascent direction to obtain a third image; processing the first predicted label and the third predicted label obtained by processing the third image by the neural network model through the first loss function to obtain a second loss; and training the neural network model based on the second loss. Thus, the data of the model decision boundary is enhanced by gradient attack, and the enhanced data is used for auxiliary training, thereby avoiding the trained model from paying too much attention to irrelevant information or local information in the image, thereby improving the performance of the model, such as generalization and robustness.

[0033] In some possible implementations, the method further includes: binarizing the first image to obtain a first true value label, the first true value label being used to characterize at least one of an edge in the first image or a location where a gradient change is greater than a preset value; processing the first true value label and at least one feature map extracted from the first image by the neural network model based on a second loss function to obtain a third loss; and training the neural network model based on the third loss. Thus, by utilizing some detail information in the image, auxiliary training of the neural network model is completed. Since some detail information of the image can be obtained after binarization, and a loss can be obtained using this detail information, and the model is trained using this loss. Therefore, the trained model can pay more attention to the detail information in the image. This improves the image processing effect of the model.

[0034] In a sixth aspect, the present application provides a neural network model training device, comprising: an encoder, a first decoder, a second decoder and a loss calculator.

[0035] The encoder is used to obtain a first image and perform feature extraction on the first image to obtain n first feature maps, where n ≥ 2, and different first feature maps have different scales. The first decoder is used to upsample k first feature maps of the n first feature maps to obtain k second feature maps, where k ≥ 1, and the scale of each feature map of the k first feature maps is larger than the scale of the remaining (nk) first feature maps of the n first feature maps. The first decoder is also used to fuse the k second feature maps with the (nk) first feature maps to obtain a first predicted label. The second decoder is used to fuse the k first feature maps with the feature map fused by the first decoder to obtain a second predicted label. The loss calculator is used to obtain the loss corresponding to the first image based on the first predicted label, the second predicted label, and the sample label of the first image. The loss calculator is also used to train the encoder, the first decoder, and the second decoder based on the loss corresponding to the first image.

[0036] Exemplarily, the components included in the neural network model training device can be understood as instances (such as threads, processes, etc.) for executing a certain code. For example, an encoder can be understood as a thread or process for executing code for acquiring an image and extracting features from the image.

[0037] In some possible implementations, when the loss calculator obtains the loss corresponding to the first image based on the first predicted label, the second predicted label and the sample label of the first image, it is specifically used to: based on the first loss function, process the first predicted label and the sample label of the first image to obtain the first loss; based on the second loss function, process the second predicted label and the sample label of the first image to obtain the second loss; based on the first loss and the second loss, obtain the loss corresponding to the first image.

[0038] In some possible implementations, the neural network model training apparatus further includes a detail label generator, wherein the detail label generator is configured to perform binarization processing on the first image to obtain a first true value label, wherein the first true value label is configured to characterize an edge in the first image.

[0039] The loss calculator is further configured to process the first true value label and at least one feature map of the n first feature maps based on a third loss function to obtain a third loss. The loss calculator is further configured to train the encoder, the first decoder, and the second decoder based on the third loss.

[0040] In some possible implementations, the neural network model training apparatus further includes an adversarial example enhancer. The adversarial example enhancer is configured to obtain a first gradient associated with the loss corresponding to the first image based on the loss corresponding to the first image, and to add a perturbation to the first image based on the first gradient and in a gradient ascent direction to obtain a second image.

[0041] The loss calculator is also used to train the encoder, the first decoder, and the second decoder using sample labels of the second image and the first image.

[0042] In some possible implementations, the neural network model training device is deployed in a cloud server. After the loss calculator completes training the encoder, the first decoder, and the second decoder, the encoder and the first decoder can be used together to perform semantic segmentation on the image obtained by the cloud server, or the encoder and the second decoder can be used together to perform semantic segmentation on the image obtained by the cloud server.

[0043] In some possible implementations, the neural network model training device is deployed in a terminal device. After the loss calculator completes training of the encoder, the first decoder, and the second decoder, the encoder and the first decoder can be used together to perform semantic segmentation on an image captured by the terminal device. For example, the terminal device can be a vehicle.

[0044] In some possible implementations, the neural network model training device is deployed in a cloud server. After the loss calculator completes training of the encoder, the first decoder, and the second decoder, the encoder and the first decoder are used together to perform semantic segmentation on an image captured by the terminal device. For example, the terminal device may be a vehicle.

[0045] In some possible implementations, when the encoder and the first decoder are jointly used to perform semantic segmentation on a third image acquired by a cloud server or a terminal device, the encoder is specifically configured to extract features from the third image to obtain n third feature maps, where n ≥ 2, and different third feature maps have different scales. The first decoder is specifically configured to fuse the n third feature maps to obtain a third predicted label, and to output the third predicted label.

[0046] In some possible implementations, the neural network model training device also includes: an adversarial sample enhancer. The adversarial sample enhancer is used to perform data enhancement on the third image to obtain a fourth image. The loss calculator is further used to process the third predicted label and the fourth predicted label through a fourth loss function to obtain a fourth loss, and the fourth predicted label is obtained by processing the fourth image by the encoder and the first decoder. The adversarial sample enhancer is also used to add perturbations to the third image along the gradient ascent direction based on the fourth loss to obtain a fifth image. The loss calculator is also used to process the third predicted label and the fifth predicted label through a fourth loss function to obtain a fifth loss, and the fifth predicted label is obtained by processing the fifth image by the encoder and the first decoder. The loss calculator is also used to update the parameters in the encoder and the first decoder based on the fifth loss.

[0047] In some possible implementations, the neural network model training device further includes an adversarial sample enhancer. The adversarial sample enhancer is configured to perform data augmentation on the third image to obtain a fourth image. The loss calculator is further configured to process the third predicted label and the fourth predicted label using a fourth loss function to obtain a fourth loss, where the fourth predicted label is obtained by processing the fourth image by the encoder and the first decoder. The adversarial sample enhancer is further configured to add perturbations to the third image along a gradient ascent direction based on the fourth loss to obtain a fifth image. The loss calculator is further configured to process the third predicted label and the fifth predicted label using the fourth loss function to obtain a fifth loss, where the fifth predicted label is obtained by processing the fifth image by the encoder and the first decoder. The loss calculator is further configured to process the third predicted label and the sixth predicted label using a fifth loss function to obtain a sixth loss, where the sixth predicted label is obtained by processing the fifth image by the encoder and the second decoder. The loss calculator is further configured to update parameters in the encoder and the first decoder based on the fifth and sixth losses.

[0048] In some possible implementations, the neural network model training device further includes a detail label generator. The detail label generator is configured to binarize the third image to obtain a second ground-truth label, where the second ground-truth label is used to represent an edge in the third image. A loss calculator is configured to process the second ground-truth label and at least one feature map from the n third feature maps based on a sixth loss function to obtain a seventh loss. The loss calculator is further configured to update parameters in the encoder and the first decoder based on the seventh loss.

[0049] In a seventh aspect, the present application provides an image processing device, which may include: an encoder, a first decoder, and a second decoder.

[0050] The encoder may be used to acquire a first image, and perform feature extraction on the first image to obtain n first feature maps, where n≥2, and different first feature maps have different scales.

[0051] The first decoder can be used to fuse the n first feature maps to obtain a first predicted label, and output the first predicted label.

[0052] The second decoder can be used to optimize the first decoder when training the first decoder, wherein the scale of the feature map processed by the second decoder is larger than the scale of the feature map processed by the first decoder.

[0053] Exemplarily, the components included in the image processing apparatus can be understood as instances (such as threads, processes, etc.) for executing a certain code. For example, an encoder can be understood as a thread or process for executing code for acquiring an image and performing feature extraction on the image.

[0054] In some possible implementations, the image processing device may be deployed in a vehicle. In this case, the first image is an image captured while the vehicle is driving. The first predicted label includes one or more of the following: lane markings, guide arrows, road traffic markings, road traffic signs, crosswalks, guide strips, and poles on the road.

[0055] In some possible implementations, the apparatus further includes: an adversarial sample enhancer and a loss calculator.

[0056] The adversarial sample enhancer can be used to perform data augmentation on the first image to obtain a second image. The loss calculator can be used to process the first predicted label and the second predicted label using a first loss function to obtain a first loss, where the second predicted label is obtained by processing the second image using the encoder and the first decoder.

[0057] In addition, the adversarial sample enhancer can be further configured to add a perturbation to the first image along a gradient ascent direction based on the first loss to obtain a third image. The loss calculator can be further configured to process the first predicted label and the third predicted label using the first loss function to obtain a second loss, where the third predicted label is obtained by processing the third image using the encoder and the first decoder; and update parameters in the encoder and the first decoder based on the second loss.

[0058] In some possible implementations, the apparatus further includes: an adversarial sample enhancer and a loss calculator.

[0059] The adversarial sample enhancer can be used to perform data augmentation on the first image to obtain a second image. The loss calculator can be used to process the first predicted label and the second predicted label using a first loss function to obtain a first loss, where the second predicted label is obtained by processing the second image using the encoder and the first decoder.

[0060] In addition, the adversarial sample enhancer can also be used to add perturbations to the first image along the gradient ascent direction based on the first loss to obtain a third image. The loss calculator can also be used to process the first predicted label and the third predicted label using the first loss function to obtain a second loss, where the third predicted label is obtained by processing the third image by the encoder and the first decoder. The loss calculator can also be used to process the first predicted label and the fourth predicted label using the second loss function to obtain a third loss, where the fourth predicted label is obtained by processing the third image by the encoder and the second decoder. The loss calculator can also be used to update parameters in the encoder and the first decoder based on the second loss and the third loss.

[0061] In some possible implementations, the device further includes: a detail tag generator and a loss calculator.

[0062] The detail label generator may be configured to binarize the first image to obtain a true label, where the true label is used to characterize edges in the first image. The loss calculator may be configured to process the true label and at least one feature map from the n first feature maps based on a third loss function to obtain a fourth loss. The loss calculator may also be configured to update parameters in the encoder and the first decoder based on the fourth loss.

[0063] In an eighth aspect, the present application provides a terminal device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation of the first aspect, or to execute the method described in the second aspect or any possible implementation of the second aspect. Exemplarily, the terminal device may be a vehicle.

[0064] In the ninth aspect, the present application provides a server comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory; wherein, when the programs stored in the memory are executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect, or to execute the method described in the second aspect or any possible implementation of the second aspect, or to execute the method described in the fifth aspect or any possible implementation of the fifth aspect.

[0065] In the tenth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect, or executes the method described in the second aspect or any possible implementation of the second aspect, or executes the method described in the fifth aspect or any possible implementation of the fifth aspect.

[0066] In an eleventh aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect, or executes the method described in the second aspect or any possible implementation of the second aspect, or executes the method described in the fifth aspect or any possible implementation of the fifth aspect.

[0067] It can be understood that the beneficial effects of the third, fourth, sixth to eleventh aspects mentioned above can be found in the relevant descriptions in the first, second or fifth aspects mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] FIG1 is a schematic diagram of the architecture of a model training system provided in an embodiment of the present application;

[0069] FIG2 is a schematic structural diagram of a decoder provided in an embodiment of the present application;

[0070] FIG3 is a flow chart of an image processing method provided in an embodiment of the present application;

[0071] FIG4 is a schematic diagram of a process for updating a neural network model provided by an embodiment of the present application;

[0072] FIG5 is a schematic diagram of a process for updating a neural network model provided by an embodiment of the present application;

[0073] FIG6 is a flow chart of a model training method provided in an embodiment of the present application;

[0074] FIG7 is a schematic diagram of a process for auxiliary training of a model provided by an embodiment of the present application;

[0075] FIG8 is a schematic diagram of a process for auxiliary training of a model provided by an embodiment of the present application;

[0076] FIG9 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0077] FIG10 is a schematic structural diagram of a model training device provided in an embodiment of the present application;

[0078] FIG11 is a schematic structural diagram of a neural network model training device provided in an embodiment of the present application;

[0079] FIG12 is a schematic structural diagram of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0081] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0082] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0083] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0084] Generally, a neural network model can be trained on a server first. Then, the number of residual blocks, the depth and height of the convolution kernels, the width of the feature maps, and the size of the pooling kernels in the neural network model's backbone are streamlined to achieve lightweighting of the neural network model. Finally, the streamlined backbone network and the decoding portion of the neural network model used for semantic segmentation can be deployed on the device side (such as a mobile phone or an in-vehicle terminal). While this approach can lightweight the neural network model to a certain extent, since it only lightweights the backbone network of the neural network model and does not lightweight the decoding portion of the neural network model used for semantic segmentation, the computing speed on the device side is still not ideal. Furthermore, since it only streamlines the backbone network without changing its structure, the streamlined backbone network is not necessarily the optimal solution. Furthermore, since the latency of the model on the server side and the latency on the device side are not strongly correlated, the streamlined backbone network may not necessarily be well adapted to the device side.

[0085] In view of this, an embodiment of the present application provides a method for applying a lightweight model on the end side while taking into account both real-time and precision performance. In this method, during the model training process, a decoder with a larger computational load is used to optimize a decoder with a smaller computational load, so that the decoder with a smaller computational load can achieve a better decoding effect under a smaller computational load. At the same time, when performing image processing on the end side, a decoder with a smaller computational load is used for decoding, but a decoder with a larger computational load is not used. In this way, by decoupling the decoder with a larger computational load from the decoder with a smaller computational load, the neural network model can be trained to perform image processing using the first decoder with a smaller computational load, thereby achieving lightweighting of the neural network model while ensuring the image processing effect during the image processing process. In this way, a lightweight model can be deployed on the end side, thereby improving the image processing effect and efficiency on the end side, and thus reducing the latency and memory consumption on the end side.

[0086] For example, Figure 1 shows the architecture of a model training system. As shown in Figure 1 , the model training system 100 may include: an encoder 110 , a decoder 120 , a detail label generator 130 , a loss calculator 140 , and an adversarial sample enhancer 150 .

[0087] Among them, the encoder 110 is mainly used to extract features from the images contained in the training samples to obtain feature maps of different scales (shapes). For example, after the encoder 110 extracts features from an image, it can obtain feature maps of scales such as 4 times, 8 times, 16 times, and 32 times. In some embodiments, the scale (shape) can refer to: the calculation result of a certain layer in the neural network, which is a tensor (tensor) in the calculation process. The scale (shape) of the tensor indicates the arrangement (dimension) of the data in the memory and the data length of each dimension, such as: 1) After a general 3-channel image is converted into a tensor, its scale is [H, W, C], where H is the image height dimension, W is the width dimension, and C is the number of channels (3); 2) After the image is calculated by the neural network, the tensor expression of the Nth feature map is output, and its scale (shape) is transformed into [H / 16, W / 16, C1].

[0088] Decoder 120 is primarily responsible for fusing feature maps of different scales extracted by encoder 110 to obtain a predicted label for the image. Decoder 120 may include a primary decoding module 121 and an auxiliary decoding module 122. Primary decoding module 121 is primarily responsible for fusing feature maps of different scales. Auxiliary decoding module 122 is primarily responsible for fusing at least one large-scale feature map with the output of primary decoding module 121 to obtain another predicted label for the image.

[0089] In this embodiment, the main decoding module 121 may include an attention fusion module (AFM) 1211 and a spatial pyramid fusion multi-attention module (SPFMA) 1212. The attention fusion module 1211 is mainly used to perform cross-attention fusion on multiple small-scale (such as 16 times and 32 times, etc.) feature maps, thereby performing attention fusion between feature maps of different scales to obtain better features. Among them, the attention fusion module 1211 can use the global information of the small-scale feature map as the attention coefficient to perform inter-layer attention with the large-scale feature map to better utilize the global information of the small-scale feature map, thereby fusing more shallow features in the image. The spatial pyramid multi-attention module 1212 is mainly used to perform spatial pyramid fusion on the results output by the attention fusion module 1211 and feature maps of other scales (such as 4 times, 8 times, etc.) other than the feature map processed by the attention fusion module 1211, so as to improve the semantic segmentation effect, thereby obtaining a predicted label for the image in the training sample. Exemplarily, the spatial pyramid multi-level attention module 1212 can upsample the feature map of at least one scale other than the feature map processed by the attention fusion module 1211 (such as the feature map of the largest scale, etc.) to reduce the scale of the feature map of that scale, thereby reducing the computational complexity of the main decoding module 121.

[0090] The auxiliary decoding module 122 may include an attention refinement module (ARM) module 1221 and a feature fusion module (FFM) 1222. The attention optimization module 1221 is mainly used to optimize at least one large-scale feature map through a self-attention mechanism to pay attention to the more important areas in the feature map. The scale of the feature map processed by the attention optimization module 1221 is greater than or equal to the largest scale of the feature map processed by the main decoding module 121. The feature fusion module 1222 is mainly used to fuse the results output by the attention optimization module 1221 and the results output by the main decoding module 121 to obtain a predicted label for the image in the training sample.

[0091] For ease of understanding, the working process of the decoder 120 is described below with an example.

[0092] As shown in Figure 2, if the encoder 110 extracts feature maps at four scales, these four scales are 4x, 8x, 16x, and 32x, respectively. During the processing of the decoder 120, the main decoding module 121 within it can first use the attention fusion module 1211 to fuse the feature maps at the 16x scale and the 32x scale to obtain a feature map at the 16x scale; then, the spatial pyramid multi-level attention module 1212 is used to upsample the feature map at the 4x scale to obtain a feature map at the 8x scale; finally, the spatial pyramid multi-level attention module 1212 is used to perform spatial pyramid fusion on the two feature maps at the 8x scale and the feature map at the 16x scale (i.e., the feature map output by the attention fusion module 1211) to obtain a predicted label. In addition, the auxiliary decoding module 122 within the decoder 120 can first use the attention optimization module 1221 to optimize the feature map at the 4x scale; then, the feature fusion module 1222 is used to fuse the processed result with the result output by the main decoding module 121 to obtain another predicted label.

[0093] The detail label generator 130 is mainly used to generate detail labels for images in the training samples to supervise the shallow features of the images in the training samples. In this embodiment, the detail label generator 130 can binarize the image p in the training sample by convolution operators such as Sobel to obtain detail labels. In this way, the clarity of the image segmentation edge and the segmentation effect of small targets in the distance can be improved by the detail label. At the same time, self-supervision can be achieved without labels, reducing dependence on manual annotation. In some embodiments, the detail label can also be called a true value label, which can be used to characterize the edge in the image. Exemplarily, the edge in the image can refer to, but is not limited to, the edge of the object to be semantically segmented in the image, such as the gradient of the edge of the object. Exemplarily, the edge of the image can be, but is not limited to, obtained from the position where the gradient change in the image is greater than a preset value.

[0094] The loss calculator 140 is mainly used to calculate the loss during model training. The loss calculator 140 may include calculation modules 141, 142 and 143.

[0095] The calculation module 141 may process the predicted labels output by the main decoding module 121 in the decoder 120 and the sample labels of the images in the training samples through a loss function (such as cross entropy, dice loss, etc.) to obtain a loss.

[0096] The calculation module 142 can process the predicted label output by the auxiliary decoding module 122 in the decoder 120 and the sample label of the image in the training sample through a loss function (such as cross entropy, dice loss, etc.) to obtain a loss.

[0097] The calculation module 143 can process the detail labels output by the detail label generator 130 and the feature map of at least one scale output by the encoder 110 through a loss function (such as cross entropy, dice loss, etc.) to obtain a loss. The loss can represent the loss of at least one of the edges corresponding to the image in the training sample or the positions where the gradient change is greater than a preset value. The loss is used to assist in training, so that during the training process, attention can be paid to the details in the image (such as the edges of the objects to be semantically segmented in the image), thereby improving the segmentation accuracy of the details in the image.

[0098] The adversarial sample enhancer 150 is mainly used to determine the gradient that needs to be adjusted during the model training process based on the relationship between the loss and the gradient during the model training process and the loss calculated by the loss calculator 140; and, using the gradient and in the direction of the gradient rise, add perturbations to the image p in the training sample to obtain a new image, thereby completing the enhancement of the data in the training sample and improving the generalization and adversarial robustness of the model. In other words, the adversarial sample enhancer 150 is mainly used to: during the training process, based on the gradient signal of the model training, obtain a specific perturbation, and use the perturbation to perform data enhancement on the image p in the training sample to make the model decision boundary more robust, thereby improving the generalization and adversarial robustness of the model. In addition, when the trained model is deployed on the end side, the enhanced data obtained by the adversarial sample enhancer 150 can also be used for comparative learning to complete the self-supervised training on the end side. For example, when the perturbation is determined by the gradient, the perturbation corresponding to the currently calculated gradient can be determined by the correspondence between the pre-set gradient and the perturbation, or the perturbation can be calculated by the following formula:

[0099]

[0100] Where η is the perturbation; ε is the coefficient, which can be set in advance; sign is the sign function, when the gradient of J is greater than 0, the result is 1, and when it is less than 0, the result is -1; is the gradient of the loss function J; θ is the model parameter; x is the prediction result of the image input to the model; y is the label of the image input to the model. The perturbation η can be a matrix that includes the perturbation corresponding to each pixel in the image input to the model.

[0101] After obtaining the perturbation corresponding to each pixel, when adding perturbation to the image, for any pixel, the pixel value of the pixel can be added to the corresponding perturbation to complete the perturbation at that pixel. After adding perturbation to each pixel, a new image can be obtained.

[0102] It should be understood that model parameters are determined by the model's loss and optimization strategy, while the gradient determines the direction of parameter updates. In an image, areas with large gradient changes are the areas where the model primarily focuses when making decisions. If excessive focus is placed on irrelevant or local information during the initial training phase, the model can become trapped in a local optimum. Model parameters are updated during each iteration, so the model's decision boundary changes dynamically during training. Therefore, during training, gradient attacks can be used to augment the data at the model's decision boundary to improve model generalization and robustness.

[0103] After obtaining the new image, the adversarial example enhancer 150 may input the new image into the encoder 110 and process it through the decoder 120 to obtain a prediction result for the new image. In addition, after obtaining the new image, the adversarial example enhancer 150 may also input the new image into the detail label generator 130 to obtain a detail label for the new image.

[0104] Furthermore, the loss calculator 140 can process the predicted labels corresponding to the image processed by the adversarial sample enhancer 150 and the sample labels of the image based on the loss function to obtain the corresponding loss. At this time, the losses obtained can also be two, namely the loss corresponding to the prediction result output by the main decoding module 121 and the loss corresponding to the prediction result output by the auxiliary decoding module 122. Then, the loss calculator 140 can also calculate the consistency loss for the loss obtained by the main decoding module 121 processing the image p in the training sample and the loss obtained by the main decoding module 121 processing the image processed by the adversarial sample enhancer 150, so as to make the two prediction results as consistent as possible, so that the model can maintain stable and consistent prediction results under different perturbation inputs. Exemplarily, the function used to calculate the consistency loss can be relative entropy (Kullback-Leibler, KL); the input of the loss function can be: the label predicted by the model before data enhancement and the label predicted by the model after data enhancement.

[0105] At the same time, the loss calculator 140 can also calculate the consistency loss of the loss obtained by the auxiliary decoding module 122 processing the image p in the training sample and the loss obtained by the auxiliary decoding module 122 processing the image processed by the adversarial sample enhancer 150, so as to make the two prediction results as consistent as possible.

[0106] In addition, the loss calculator 140 may also process the detail labels corresponding to the image processed by the adversarial sample enhancer 150 and the feature map at at least one scale corresponding to the image based on the loss function to obtain the loss of certain details in the image (e.g., at least one of an edge in the image or a location where the gradient change is greater than a preset value). The loss calculator 140 may then calculate a consistency loss between the loss of certain details in the image calculated this time and the loss of details obtained by processing the images in the training sample, so as to ensure that the two results are as consistent as possible.

[0107] Finally, the loss calculator 140 can calculate the various losses it has calculated to obtain the final loss in the model training process. In some embodiments, the loss calculator 140 can add up the various losses it has calculated and average them to obtain the final loss. In addition, the loss calculator 140 can also obtain the final loss by weighted summing up the various losses. Exemplarily, when calculating the final loss, the losses used may include: the loss calculated by the loss calculator 140 through the decoder 120 to process the image p in the training sample and the output of the predicted label, and the loss calculator 140 calculated by the detail label generator 130 to process the image p in the training sample and the various consistency losses calculated by the loss calculator 140 through the adversarial sample enhancer 150.

[0108] After the final loss is obtained by the loss calculator 140 , at least the parameters in the encoder 110 and the decoder 120 can be updated using the loss to complete the training of the encoder 110 and the decoder 120 .

[0109] The above is an introduction to the model training system provided in the embodiment of the present application. It is understandable that the model training system can be configured in a device with model training capabilities, such as a cloud server or a terminal device, and the specific configuration can be determined according to the actual situation.

[0110] After the encoder 110 and the decoder 120 are trained by the model training system shown in FIG1 , the main decoding module 121 in the encoder 110 and the decoder 120 can be configured on a terminal device (such as a mobile phone, a vehicle, etc.). In this way, the encoder 110 and the main decoding module 121 can be used for reasoning on the end side. Since the embodiment of the present application adopts a decoupled design of the main decoding module 121 and the auxiliary decoding module 122, the two modules can be jointly optimized during the model training phase, and during reasoning, only the main decoding module 121 for fusion decoding of small-scale features can be used, thereby reducing the amount of computation on the end side, thereby reducing the latency and memory consumption on the end side.

[0111] In addition, since the parameters in the main decoding module 121 are updated using the loss corresponding to the auxiliary decoding module 122, which has a larger computational load, during the model training process, that is, the auxiliary decoding module 122 is used to assist in training the main decoding module 121, so that the two can be integrated during the training phase, the trained main decoding module 121 can also achieve the optimal decoding effect with a smaller computational load, thereby improving the accuracy of semantic segmentation. In addition, since some details such as edges of the image are also paid attention to by the detail label generator 130 during the model training process, the details of the image can also be paid attention to when the main decoding module 121 is used on the end side, thereby improving the accuracy and effect of semantic segmentation. In some embodiments, the main decoding module 121 in the encoder 110 and decoder 120 deployed on the end side can be saved on the end side in the form of a binary file, and an AI framework can be configured on the end side, through which the binary file can be automatically loaded to achieve semantic segmentation of the image obtained on the end side.

[0112] In addition, to further lightweight the model deployed on the end side, in embodiments of the present application, the encoder 110 can also be lightweighted. For example, model distillation or model migration can be performed on the encoder 110 to obtain a lightweight encoder. The network structure of the encoder 110 can also be directly designed as a lightweight result before model training. Alternatively, a network structure can be automatically searched and found within a manually set search range through reinforcement learning or other methods, and this network structure can be used as the structure of the encoder 110, thereby completing the lightweighting of the encoder 110.

[0113] Furthermore, in order to improve the accuracy and effect of semantic segmentation on the end side, the above-mentioned model training system 100 can also be deployed on the end side so that the end side can use the acquired images to update the model in the model training system 100.

[0114] When updating the model in the model training system 100 on the end-side, since the images acquired on the end-side are often unlabeled, self-supervised training can be used to alleviate reliance on data labeling. Specifically, a perturbation can be randomly generated and used to process the image acquired on the end-side to obtain a new image. These two images can then be input into the decoder 120 to obtain corresponding predicted labels. The calculation module 141 in the loss calculator 140 can then use a loss function, such as a pixel-wise contrastive loss, to process the prediction result corresponding to the image acquired on the end-side and the predicted label corresponding to the image obtained by adding the perturbation to obtain a loss. Furthermore, the calculation module 142 in the loss calculator 140 can also use a loss function, such as a pixel-wise contrastive loss, to process the predicted label corresponding to the image acquired on the end-side and the predicted label corresponding to the image obtained by adding the perturbation to obtain another loss. Then, the adversarial sample enhancer 150 can calculate a gradient through these two losses and use the gradient to obtain the corresponding perturbation. Then, the adversarial sample enhancer 150 can use the perturbation to process the image obtained by the end side to obtain another new image. Then, the decoder 120 can be used to predict the newly obtained image. Finally, the loss calculator 140 can use a loss function, such as a pixel-wise contrastive loss function (pixel-wise contrastive loss), to process the predicted label corresponding to the image obtained by the end side and the predicted label corresponding to the image output by the adversarial sample enhancer 150 to obtain a loss. Finally, the loss obtained by the loss calculator 140 can be used to update the model parameters in the encoder 110 and / or decoder 120. This completes the update of the model in the model training system 100. It should be understood that for other processes of updating the model on the end side, please refer to the relevant description in the aforementioned Figure 1, which will not be repeated here. Of course, when the terminal side updates the model, the loss output by the calculation module 143 in the loss calculator 140 can also be used. Please refer to the above description for details, which will not be repeated here.

[0115] In some embodiments, when the client updates the model, if only the primary decoding module 121 in the decoder 120 is used for prediction, the predicted label obtained by the decoder 120 for each image is a single label. If both the primary decoding module 121 and the auxiliary decoding module 122 in the decoder 120 are used for prediction, the predicted labels obtained by the decoder 120 for each image are two labels. In this case, when calculating the loss, the loss calculator 140 can use the loss to process different results output by the same module to obtain the loss when using the module for prediction. In this way, the loss calculator 140 can calculate two losses for the decoder 120.

[0116] In some embodiments, when updating the model, the client can use the detail label generator 130 in the trained model to binarize the image used to update the model, so that the model can pay more attention to details in the image (such as edges in the image or locations where the gradient change is greater than a preset value). The model is then used to extract features of the image used to update the model. Next, the calculation module 143 in the model processes the binarization results and the features extracted by the model to obtain a corresponding loss. Finally, the model is updated using this loss.

[0117] In some embodiments, the model training system 100 described in FIG1 can be deployed on a cloud server. In this case, after completing the model training on the cloud server, the main decoding module 121 in the encoder 110 and decoder 120 obtained by training can be deployed on the terminal device. This enables cloud-side training and device-side reasoning. In addition, in order to enable the update of the model on the device side, the cloud server can also send the trained model to the device side so that the device side can perform self-supervised training. This enables the three capabilities of cloud-side training, device-side reasoning, and device-side self-supervised training.

[0118] In addition, model training and inference can also be performed on the cloud server, that is, semantic segmentation of the image can be performed. In this case, semantic segmentation can be performed using the main decoding module 121 in the encoder 110 and the decoder 120, or using the auxiliary decoding module 122 in the encoder 110 and the decoder 120.

[0119] In some embodiments, the model training system 100 shown in FIG1 can be deployed on a terminal device. In this case, after model training is completed on the terminal device, the trained encoder 110 and the main decoding module 121 in the decoder 120 can be used to perform semantic segmentation. This enables on-device training and inference.

[0120] In addition, the device can also perform self-supervised training on the model to update the model. This enables three capabilities: on-device training, on-device reasoning, and on-device self-supervised training.

[0121] The above is an introduction to the model training system 100 provided in an embodiment of the present application. Next, based on the content described above, an image processing method provided in an embodiment of the present application is introduced. It is understood that the content described in this method can be, but is not limited to, reference to part or all of the content described above.

[0122] For example, FIG3 shows an image processing method. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. The method can be executed on the end side (such as a mobile phone, vehicle, etc.) or on the cloud side (such as a cloud server, etc.). As shown in FIG3, the image processing method may include the following steps:

[0123] S301: Acquire a first image.

[0124] In this embodiment, the first image may be, but is not limited to, captured in real time by an image capture device such as a camera. For example, when the method is performed by a vehicle, the first image may be an image of the road captured in real time by the vehicle's image capture device. In other embodiments, the first image may also be an image previously captured by the image capture device, or an image periodically captured by the image capture device.

[0125] S302. Perform feature extraction on the first image through an encoder in the neural network model to obtain n first feature maps, where n≥2, and different first feature maps have different scales.

[0126] In this embodiment, the first image can be feature extracted by an encoder in the neural network model to obtain n (n ≥ 2) first feature maps. Different first feature maps have different scales (shapes). Exemplarily, the neural network model can be a model trained by the model training system 100 described in FIG. 1 . The encoder in the neural network model can be the encoder 110 described in FIG. 1 .

[0127] S303. Fusing n first feature maps through a first decoder in a neural network model to obtain a first prediction label, wherein a second decoder is further configured in the neural network model, and the second decoder is used to optimize the first decoder when training the neural network model, and the scale of the feature map processed by the second decoder is larger than the scale of the feature map processed by the first decoder.

[0128] In this embodiment, after obtaining n first feature maps, the n first feature maps can be fused through the first decoder in the neural network model to obtain a first prediction label. Among them, the neural network model is also configured with a second decoder. The second decoder is used to optimize the first decoder when training the neural network model, and the scale of the feature map processed by the second decoder is larger than the scale of the feature map processed by the first decoder. Exemplarily, the first decoder can be the main decoding module 121 in the decoder 120 described in the aforementioned Figure 1, and the second decoder can be the auxiliary decoding module 122 in the decoder 120 described in the aforementioned Figure 1. For the process of feature map fusion by the first decoder, please refer to the description of the main decoding module 121 in the aforementioned Figure 1 for details, which will not be repeated here.

[0129] S304: Output a first predicted label.

[0130] In this embodiment, after obtaining the first predicted label, the semantic segmentation of the first image is completed, and the first predicted label can be output at this time. In some embodiments, the first predicted label can be understood as the result of semantic segmentation of the first image.

[0131] Thus, the processing of the first image is completed. Since only the first decoder in the neural network model is used for decoding during the image processing process, and the scale of the image processed by the first decoder is lower than the scale of the image processed by the second decoder, the computational complexity of the first decoder is smaller than that of the second decoder. At the same time, since the second decoder is used to optimize the first decoder when training the neural network model, the first decoder can achieve better decoding effects with a smaller computational complexity. In this way, by decoupling the first decoder and the second decoder, the image processing effect is guaranteed while the neural network model is lightweight during the image processing process. This can improve the image processing effect and efficiency on the end side, thereby reducing the latency and memory consumption on the end side.

[0132] In some embodiments, executing an event through a module in a neural network model can be understood as executing the event by executing code associated with the module in the neural network model. In this case, a module in the neural network model can be understood, but is not limited to, as a string of code. For example, extracting features from a first image through an encoder in a neural network model can be understood as extracting features from the first image by executing code associated with the encoder in the neural network model.

[0133] In some embodiments, when the image processing method described in FIG. 3 is executed by a vehicle, the first image acquired in S301 may be an image captured while the vehicle is driving. For example, the vehicle may capture a video while driving and extract a frame of image from the video as the first image; or the vehicle may directly capture an image while driving.

[0134] At this time, the first predicted label obtained in S303 may include one or more of the following: lane lines, guide arrows, road traffic markings, road traffic signs, pedestrian crossing lines, guide strips, and pole-shaped objects on the road.

[0135] Furthermore, in S304, the vehicle may display the first predicted label on the front windshield of the vehicle or on another display device in the vehicle. In this way, the driver in the vehicle can intuitively understand the current situation on the road where the vehicle is located. In addition, the arrow of the currently used navigation can also be rendered to form real-time navigation on the road (e.g., intersection, fork, etc.).

[0136] In some embodiments, when the image processing method described in FIG. 3 is executed by a mobile phone, the first image acquired in S301 may be an image captured by a user using the mobile phone.

[0137] At this time, the first predicted label obtained in S303 may include fixed objects in the environment.

[0138] Furthermore, in S304, the first predicted label may be input into a model for reconstructing a scene to complete the reconstruction of the virtual scene, thereby realizing immersive virtual reality (VR).

[0139] In some embodiments, in the method described in FIG3 , the neural network model can also be updated. In some embodiments, the process of updating the neural model can be understood as, but not limited to, the process described in FIG3 , when the trained model is deployed to the end side, and the end side (such as a vehicle, mobile phone, etc.) uses the adversarial sample enhancer 150 to update the model. Of course, when the trained model is deployed on the cloud side, it can also be the process of the cloud side (such as a cloud server, etc.) using the adversarial sample enhancer 150 to update the model.

[0140] Specifically, as shown in FIG4 , the process of updating the neural network model may include:

[0141] S401: Acquire a second image.

[0142] In this embodiment, the method for acquiring the second image is the same as the method for acquiring the first image in FIG. 3 , and reference may be made to the aforementioned description, which will not be repeated here.

[0143] In some embodiments, the second image may be, but is not limited to, the same as the first image, i.e., the two are the same image. In this case, after obtaining the predicted label of the image (i.e., the image obtained after semantic segmentation), the method uses the image to update the neural network model.

[0144] S402: Perform data enhancement on the second image to obtain a third image.

[0145] In this embodiment, a perturbation may be randomly generated, and the perturbation may be used to perform data enhancement on the second image, thereby obtaining a third image.

[0146] S403. Process the second predicted label obtained by processing the second image through the neural network model and the third predicted label obtained by processing the third image through the neural network model through the first loss function to obtain a first loss.

[0147] In this embodiment, the second image can be processed by the neural network model to obtain a predicted label corresponding to the second image (hereinafter referred to as the "second predicted label"), and the third image can be processed by the neural network model to obtain a predicted label corresponding to the third image (hereinafter referred to as the "third predicted label"). The process of processing the second and third images can refer to the process of processing the first image in Figure 3 above, and will not be repeated here.

[0148] Furthermore, the second predicted label and the third predicted label can be input into a preset first loss function. The two labels are processed by the first loss function to obtain a loss, namely the first loss.

[0149] S404: Based on the first loss, add perturbation to the second image along the gradient ascent direction to obtain a fourth image.

[0150] In this embodiment, based on the first loss, perturbation can be added to the second image along the gradient ascent direction to obtain a fourth image.

[0151] S405 . Process the second predicted label and the fourth predicted label obtained by processing the fourth image through the neural network model using the first loss function to obtain a second loss.

[0152] In this embodiment, after obtaining the fourth image, the fourth image can be processed by the neural network model to obtain a predicted label corresponding to the fourth image (hereinafter referred to as the "fourth predicted label"). The process of processing the fourth image can refer to the process of processing the first image in Figure 3 above, and will not be repeated here.

[0153] Furthermore, the second predicted label and the fourth predicted label can be input into a preset first loss function. The two labels are processed by the first loss function to obtain a loss, namely, a second loss.

[0154] S406. Update the neural network model based on the second loss.

[0155] In this embodiment, after obtaining the second loss, the neural network model can be updated in a direction that minimizes the second loss, for example, by updating parameters of the encoder, the first decoder, and the second decoder in the neural network model.

[0156] In this way, self-supervised training of the neural network model is achieved without labels, thereby completing the update of the neural network model.

[0157] In some embodiments, the neural network model can also be updated in the method described in FIG3 . In some embodiments, the neural network model update process can be understood as, but not limited to, the process described in FIG3 , when the trained model is deployed on the device side, where the device side uses the detail label generator 130 to update the model. Of course, when the trained model is deployed on the cloud side, the cloud side can also use the detail label generator 130 to update the model.

[0158] Specifically, as shown in FIG5 , the process of updating the neural network model may include:

[0159] S501: Acquire a fifth image.

[0160] In this embodiment, the method for acquiring the fifth image is the same as the method for acquiring the first image in FIG. 3 , and reference may be made to the aforementioned description, which will not be repeated here.

[0161] In some embodiments, the fifth image may be, but is not limited to, the same as the first image, i.e., the two are the same image. In this case, after obtaining the predicted label of the image (i.e., the image obtained after semantic segmentation), the method uses the image to update the neural network model.

[0162] S502: Binarize the fifth image to obtain a first true value label, where the first true value label is used to characterize an edge in the fifth image.

[0163] In this embodiment, the fifth image can be binarized by the detail label generator 130 in FIG. 3 to obtain a first ground-truth label (also referred to as a "detail label"). This ground-truth label can be used to characterize edges in the fifth image. For example, the edges of the fifth image can be obtained, but are not limited to, by locations in the fifth image where a gradient change is greater than a preset value.

[0164] S503. Based on the second loss function, process the first true value label and at least one feature map extracted from the fifth image through the neural network model to obtain a third loss.

[0165] In this embodiment, after the fifth image is acquired, feature extraction may be performed on the fifth image through an encoder in the neural network model to obtain at least one feature map.

[0166] Furthermore, the first true value label and the obtained at least one feature map can be input into a preset second loss function. The data is processed by the second loss function to obtain a loss, i.e., a third loss. For example, this process can be referred to the description of the calculation module 143 in FIG. 3 , and will not be repeated here.

[0167] S504: Based on the third loss, update the neural network model.

[0168] In this embodiment, after obtaining the third loss, the neural network model can be updated in a direction that minimizes the third loss, for example, by updating parameters of the encoder, the first decoder, and the second decoder in the neural network model.

[0169] Thus, the neural network model is updated using some of the image's detailed information. After binarizing the image, some detailed information about the image is obtained, and this detailed information can be used to calculate a loss, which is then used to update the model. Therefore, the updated model can pay more attention to the detailed information in the image, thereby improving the model's image processing performance.

[0170] The above is an introduction to the model training system 100 and image processing method provided in the embodiment of the present application. Next, based on the content described above, a model training method provided in the embodiment of the present application is introduced. It is understood that the content described in this method can, but is not limited to, refer to part or all of the content described above.

[0171] For example, FIG6 illustrates a model training method. It is understood that this method can be performed by any device, apparatus, platform, or device cluster with computing and processing capabilities. S601 and S602 in FIG6 can refer to the description of S301 and S302 in FIG3 , and are not repeated here. The neural network model involved in FIG6 can be the model required to be trained by the model training system 100 in FIG1 . The encoder in the neural network model involved in FIG6 can be the encoder 110 described in FIG1 . The first decoder in the neural network model involved in FIG6 can be the main decoding module 121 described in FIG1 ; the processing process using the first decoder in S603 and S604 can refer to the description of the main decoding module 121 in FIG1 , and are not repeated here. The second decoder in the neural network model involved in FIG6 can be the auxiliary decoding module 122 described in FIG1 ; the processing process using the second decoder in S605 can refer to the description of the auxiliary decoding module 122 in FIG1 , and are not repeated here.

[0172] As shown in FIG6 , the model training method may include the following steps:

[0173] S601: Acquire a first image.

[0174] S602: Perform feature extraction on the first image using an encoder in a neural network model to obtain n first feature maps, where n≥2, and different first feature maps have different scales.

[0175] S603. Upsample k first feature maps among the n first feature maps through the first decoder in the neural network model to obtain k second feature maps, k ≥ 1, wherein the scale of each feature map in the k first feature maps is larger than the scale of the remaining (nk) first feature maps in the n first feature maps.

[0176] S604: Fusing the k second feature maps and (nk) first feature maps through a first decoder to obtain a first predicted label.

[0177] S605. Fuse the k first feature maps and the feature map fused by the first decoder through the second decoder in the neural network model to obtain a second predicted label.

[0178] S606: Obtain a loss corresponding to the first image according to the first predicted label, the second predicted label, and the sample label of the first image.

[0179] In this embodiment, after obtaining the first predicted label and the second predicted label, the first predicted label and the sample label of the first image can be input into a preset first loss function to obtain a first loss, that is, based on the first loss function, the first predicted label and the sample label of the first image are processed to obtain a loss. At the same time, the second predicted label and the sample label of the first image can be input into the preset first loss function to obtain a second loss, that is, based on the first loss function, the second predicted label and the sample label of the first image are processed to obtain a loss. Finally, based on the first loss and the second loss, for example, weighted summation, the loss corresponding to the first image can be obtained.

[0180] S607: Train a neural network model based on the loss corresponding to the first image.

[0181] In this embodiment, after obtaining the loss corresponding to the first image, the neural network model can be trained with the goal of minimizing the loss.

[0182] This completes the training of the neural network model. Since the scale of the image processed by the first decoder in the neural network model is lower than the scale of the image processed by the second decoder during model training, the computational complexity of the first decoder is smaller than that of the second decoder. At the same time, the second decoder is used to optimize the first decoder, so the first decoder can achieve better decoding results with less computational complexity. Thus, by decoupling the first and second decoders, the neural network model can be trained to perform image processing using the first decoder with less computational complexity. This allows the image processing model to be lightweight while ensuring the image processing effect during the image processing process. This allows the lightweight model to be deployed on the end-side, thereby improving the image processing effect and efficiency on the end-side, and thereby reducing end-side latency and memory consumption.

[0183] In some embodiments, during the model training process shown in FIG6 , in order to enable the trained model to pay more attention to some detail information in the image (such as the edges in the image or locations where the gradient change is greater than a preset value, etc.), the detail generation tag 130 described in FIG1 may be used for auxiliary training. Specifically, as shown in FIG7 , the process of using the detail generation tag 130 described in FIG1 for auxiliary training may include the following steps:

[0184] S701: Binarize the first image to obtain a first true value label, where the first true value label is used to represent an edge in the first image.

[0185] S702: Based on a third loss function, process the first true value label and at least one feature map of the n first feature maps to obtain a third loss.

[0186] S703: Based on the third loss, train the neural network model.

[0187] The specific descriptions of S701 to S703 may refer to the description related to the detail generation tag 130 in FIG. 1 , or refer to the description in FIG. 5 , which will not be repeated here.

[0188] Thus, by utilizing some of the image's detailed information, auxiliary training of the neural network model is completed. After binarizing the image, some of the image's detailed information can be obtained, and this detailed information can be used to generate a loss, which is then used to train the model. As a result, the trained model can pay more attention to the detailed information in the image, thereby improving the model's image processing performance.

[0189] In some embodiments, during the model training process shown in FIG6 , in order to prevent the trained model from focusing too much on irrelevant information or local information in the image, thereby causing the model to fall into a local optimum, etc., the adversarial sample enhancer 150 described in FIG1 may be used for auxiliary training. Specifically, as shown in FIG8 , the process of using the adversarial sample enhancer 150 described in FIG1 for auxiliary training may include the following steps:

[0190] S801. Based on the loss corresponding to the first image, obtain a first gradient associated with the loss corresponding to the first image.

[0191] S802: Add disturbance to the first image based on the first gradient and along the gradient ascending direction to obtain a second image.

[0192] S803: Train a neural network model using sample labels of the second image and the first image.

[0193] Among them, the specific descriptions of S801 to S803 can refer to the description related to the adversarial sample enhancer 15 in the aforementioned Figure 1, or refer to the description of S404 to S406 in the aforementioned Figure 4, which will not be repeated here.

[0194] Therefore, the data on the model decision boundary is enhanced through gradient attack, and the enhanced data is used for auxiliary training, thereby avoiding the trained model from paying too much attention to irrelevant information or local information in the image, and improving the performance of the model, such as generalization and robustness.

[0195] In some embodiments, the method described in FIG6 can be performed by a cloud server (i.e., cloud side). In this case, after completing the training of the neural network model, the cloud server can use the encoder and the first decoder in the trained neural network model to perform semantic segmentation, i.e., perform image processing, or use the encoder and the second decoder in the trained neural network model to perform semantic segmentation.

[0196] In addition, the cloud server can also send the encoder and the first decoder in the trained neural network model to the terminal device to deploy the encoder and the first decoder in the trained neural network model on the terminal device, so that the terminal device can use the encoder and the first decoder for semantic segmentation.

[0197] In some embodiments, the method described in FIG6 can be performed by a terminal device. In this case, after completing the training of the neural network model, the terminal device can use the encoder and the first decoder in the trained neural network model to perform semantic segmentation. For example, the terminal device can be, but is not limited to, a vehicle.

[0198] It is understandable that the size of the sequence number of each step in the above embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. In addition, in some possible implementations, the steps in the above embodiment can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. In addition, all or part of any features in the above embodiment can be freely and arbitrarily combined without contradiction. The combined technical solution is also within the scope of this application.

[0199] Based on the image processing method in the above embodiment, an embodiment of the present application further provides an image processing device.

[0200] For example, an image processing apparatus is shown in FIG9 . As shown in FIG9 , the image processing apparatus 900 may include: an acquisition module 901 and a processing module 902 .

[0201] The acquisition module 901 is used to acquire a first image.

[0202] Processing module 902 is used to extract features from the first image through an encoder in the neural network model to obtain n first feature maps, where n ≥ 2, and different first feature maps have different scales. Processing module 902 is also used to fuse the n first feature maps through a first decoder in the neural network model to obtain a first predicted label, wherein the neural network model is also configured with a second decoder, which is used to optimize the first decoder when training the neural network model, and the scale of the feature map processed by the second decoder is larger than the scale of the feature map processed by the first decoder. Processing module 902 is also used to output the first predicted label.

[0203] In some embodiments, the image processing device is deployed in a vehicle, the first image is an image captured while the vehicle is driving, and the first predicted label includes one or more of the following:

[0204] Lane lines, guide arrows, road traffic markings, road traffic signs, pedestrian crossing lines, guide strips, and poles on the road.

[0205] In some embodiments, the acquisition module 901 is further configured to acquire a second image;

[0206] The processing module 902 is further configured to perform data enhancement on the second image to obtain a third image;

[0207] The processing module 902 is further configured to process the second predicted label obtained by processing the second image through the neural network model and the third predicted label obtained by processing the third image through the neural network model using the first loss function to obtain a first loss;

[0208] The processing module 902 is further configured to add a perturbation to the second image along a gradient ascent direction based on the first loss to obtain a fourth image;

[0209] The processing module 902 is further configured to process the second predicted label and the fourth predicted label obtained by processing the fourth image through the neural network model using the first loss function to obtain a second loss;

[0210] The processing module 902 is further configured to update the neural network model based on the second loss.

[0211] In some embodiments, the acquisition module 901 is further configured to acquire a fifth image;

[0212] The processing module 902 is further configured to perform binarization processing on the fifth image to obtain a first true value label, where the first true value label is used to represent an edge in the fifth image;

[0213] The processing module 902 is further configured to process the first true value label and at least one feature map extracted from the fifth image by the neural network model based on the second loss function to obtain a third loss;

[0214] The processing module 902 is further configured to update the neural network model based on the third loss.

[0215] It should be understood that the above-mentioned device is used to execute the image processing method described in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0216] Based on the method in the above embodiment, the embodiment of the present application also provides a model training device.

[0217] For example, a model training device is shown in FIG10 . As shown in FIG10 , the model training device 1000 may include: an acquisition module 1001 and a processing module 1002 .

[0218] The acquisition module 1001 is configured to acquire a first image.

[0219] The processing module 1002 is configured to perform feature extraction on the first image through an encoder in a neural network model to obtain n first feature maps, where n≥2, and different first feature maps have different scales.

[0220] The processing module 1002 is also used to upsample k first feature maps in the n first feature maps through the first decoder in the neural network model to obtain k second feature maps, k≥1, wherein the scale of each feature map in the k first feature maps is greater than the scale of the remaining (nk) first feature maps in the n first feature maps.

[0221] The processing module 1002 is further configured to fuse the k second feature maps and the (nk) first feature maps through a first decoder to obtain a first predicted label.

[0222] The processing module 1002 is further configured to fuse the k first feature maps and the feature map fused by the first decoder through a second decoder in the neural network model to obtain a second predicted label.

[0223] The processing module 1002 is further configured to obtain a loss corresponding to the first image according to the first predicted label, the second predicted label, and the sample label of the first image.

[0224] The processing module 1002 is further configured to train a neural network model based on the loss corresponding to the first image.

[0225] In some embodiments, when the processing module 1002 obtains the loss corresponding to the first image based on the first predicted label, the second predicted label, and the sample label of the first image, it is specifically configured to:

[0226] Processing the first predicted label and the sample label of the first image based on the first loss function to obtain a first loss;

[0227] Based on the second loss function, processing the second predicted label and the sample label of the first image to obtain a second loss;

[0228] Based on the first loss and the second loss, the loss corresponding to the first image is obtained.

[0229] In some embodiments, the processing module 1002 is further configured to:

[0230] Binarize the first image to obtain a first true value label, where the first true value label is used to characterize an edge in the first image;

[0231] Processing the first true value label and at least one feature map of the n first feature maps based on a third loss function to obtain a third loss;

[0232] Based on the third loss, the neural network model is trained.

[0233] In some embodiments, the processing module 1002 is further configured to:

[0234] Obtaining a first gradient associated with the loss corresponding to the first image based on the loss corresponding to the first image;

[0235] Based on the first gradient and in the direction of gradient ascent, adding perturbation to the first image to obtain a second image;

[0236] The neural network model is trained using the sample labels of the second image and the first image.

[0237] In some embodiments, the model training device is deployed in a cloud server. After completing the training of the neural network model, the processing module 1002 is further configured to:

[0238] Semantic segmentation is performed using an encoder and a first decoder, or semantic segmentation is performed using an encoder and a second decoder.

[0239] In some embodiments, the model training device is deployed in the terminal device, and after completing the training of the neural network model, the processing module 1002 is further used to:

[0240] Semantic segmentation is performed using the encoder and the first decoder.

[0241] In some embodiments, the model training device is deployed in a cloud server. After completing the training of the neural network model, the processing module 1002 is further configured to:

[0242] The encoder and the first decoder are deployed on the terminal device, so that the terminal device uses the encoder and the first decoder to perform semantic segmentation.

[0243] In some embodiments, the terminal device is a vehicle.

[0244] It should be understood that the above-mentioned device is used to execute the model training method described in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0245] Based on the method in the above embodiment, the present application provides a neural network model training device.

[0246] For example, a neural network model training apparatus is shown in FIG11 . As shown in FIG11 , the neural network model training apparatus 1100 may include: an encoder 1101 , a first decoder 1102 , a second decoder 1103 , and a loss calculator 1104 .

[0247] The encoder 1101 is configured to acquire a first image and perform feature extraction on the first image to obtain n first feature maps, where n ≥ 2, and different first feature maps have different scales. The first decoder 1102 is configured to upsample k first feature maps from the n first feature maps to obtain k second feature maps, where k ≥ 1, and each feature map has a larger scale than the remaining (nk) first feature maps from the n first feature maps. The first decoder 1102 is further configured to fuse the k second feature maps with the (nk) first feature maps to obtain a first predicted label. The second decoder 1103 is configured to fuse the k first feature maps with the feature map fused by the first decoder 1102 to obtain a second predicted label. The loss calculator 1104 is configured to obtain a loss corresponding to the first image based on the first predicted label, the second predicted label, and the sample label of the first image. The loss calculator 1104 is further configured to train the encoder 1101, the first decoder 1102, and the second decoder 1103 based on the loss corresponding to the first image. Exemplarily, the encoder 1101 can be the encoder 110 described in the aforementioned Figure 1, the first decoder 1102 can be the main decoding module 121 described in the aforementioned Figure 1, the second decoder 1103 can be the auxiliary decoding module 122 described in the aforementioned Figure 1, and the loss calculator 1104 can be the loss calculator 140 described in the aforementioned Figure 1.

[0248] In some embodiments, the components included in the neural network model training device can be understood as instances (such as threads, processes, etc.) for executing a certain code. For example, an encoder can be understood as a thread or process for executing code for acquiring an image and performing feature extraction on the image.

[0249] In some embodiments, when the loss calculator 1104 obtains the loss corresponding to the first image based on the first predicted label, the second predicted label and the sample label of the first image, it is specifically used to: process the first predicted label and the sample label of the first image based on the first loss function to obtain the first loss; process the second predicted label and the sample label of the first image based on the second loss function to obtain the second loss; and obtain the loss corresponding to the first image based on the first loss and the second loss.

[0250] In some embodiments, the neural network model training device further includes a detail label generator (not shown). The detail label generator is configured to binarize the first image to obtain a first true value label, which is used to characterize an edge in the first image. Exemplarily, the detail label generator may be the detail label generator 130 described in FIG. 1 .

[0251] The loss calculator 1104 is further configured to process the first true value label and at least one feature map of the n first feature maps based on a third loss function to obtain a third loss. The loss calculator 1104 is further configured to train the encoder 1101, the first decoder 1102, and the second decoder 1103 based on the third loss.

[0252] In some embodiments, the neural network model training device further includes an adversarial sample enhancer (not shown). The adversarial sample enhancer is configured to obtain a first gradient associated with the loss corresponding to the first image based on the loss corresponding to the first image, and to add a perturbation to the first image based on the first gradient and in the direction of gradient ascent to obtain a second image. Exemplarily, the adversarial sample enhancer may be the adversarial sample enhancer 150 described in FIG1 .

[0253] The loss calculator 1104 is further configured to train the encoder 1101 , the first decoder 1102 , and the second decoder 1103 using sample labels of the second image and the first image.

[0254] In some embodiments, the neural network model training device is deployed in a cloud server. After the loss calculator 1104 completes training the encoder 1101, the first decoder 1102 and the second decoder 1103, the encoder 1101 and the first decoder 1102 can be used together to perform semantic segmentation on the image obtained by the cloud server, or the encoder 1101 and the second decoder 1103 can be used together to perform semantic segmentation on the image obtained by the cloud server.

[0255] In some embodiments, the neural network model training device is deployed in a terminal device. After the loss calculator 1104 completes training of the encoder 1101, the first decoder 1102, and the second decoder 1103, the encoder 1101 and the first decoder 1102 can be used together to perform semantic segmentation on an image captured by the terminal device. For example, the terminal device can be a vehicle.

[0256] In some embodiments, the neural network model training device is deployed in a cloud server. After the loss calculator 1104 completes training of the encoder 1101, the first decoder 1102, and the second decoder 1103, the encoder 1101 and the first decoder 1102 are used together to perform semantic segmentation on an image captured by the terminal device. For example, the terminal device can be a vehicle.

[0257] In some embodiments, when the encoder 1101 and the first decoder 1102 are jointly used to perform semantic segmentation on the third image acquired by the cloud server or the terminal device, the encoder 1101 is specifically configured to extract features from the third image to obtain n third feature maps, where n ≥ 2, where different third feature maps have different scales. The first decoder 1102 is specifically configured to fuse the n third feature maps to obtain a third predicted label, and to output the third predicted label.

[0258] In some embodiments, the neural network model training apparatus further includes an adversarial sample enhancer (not shown in the figure). Exemplarily, the adversarial sample enhancer may be the adversarial sample enhancer 150 described in FIG1 .

[0259] The adversarial sample enhancer is used to perform data augmentation on the third image to obtain a fourth image. The loss calculator 1104 is further used to process the third predicted label and the fourth predicted label using a fourth loss function to obtain a fourth loss. The fourth predicted label is obtained by processing the fourth image by the encoder 1101 and the first decoder 1102. The adversarial sample enhancer is further used to add perturbations to the third image along the gradient ascent direction based on the fourth loss to obtain a fifth image. The loss calculator 1104 is further used to process the third predicted label and the fifth predicted label using the fourth loss function to obtain a fifth loss. The fifth predicted label is obtained by processing the fifth image by the encoder 1101 and the first decoder 1102. The loss calculator 1104 is further used to update the parameters in the encoder 1101 and the first decoder 1102 based on the fifth loss.

[0260] In some embodiments, the neural network model training apparatus further includes an adversarial sample enhancer (not shown in the figure). Exemplarily, the adversarial sample enhancer may be the adversarial sample enhancer 150 described in FIG1 .

[0261] The adversarial example enhancer is configured to perform data augmentation on the third image to obtain a fourth image. The loss calculator 1104 is further configured to process the third predicted label and the fourth predicted label using a fourth loss function to obtain a fourth loss. The fourth predicted label is obtained by processing the fourth image through the encoder 1101 and the first decoder 1102. The adversarial example enhancer is further configured to add perturbations to the third image along the gradient ascent direction based on the fourth loss to obtain a fifth image. The loss calculator 1104 is further configured to process the third predicted label and the fifth predicted label using the fourth loss function to obtain a fifth loss. The fifth predicted label is obtained by processing the fifth image through the encoder 1101 and the first decoder 1102. The loss calculator 1104 is further configured to process the third predicted label and the sixth predicted label using a fifth loss function to obtain a sixth loss. The sixth predicted label is obtained by processing the fifth image through the encoder 1101 and the second decoder 1103. The loss calculator 1104 is further configured to update parameters in the encoder 1101 and the first decoder 1102 based on the fifth and sixth losses.

[0262] In some embodiments, the neural network model training device further includes: a detail label generator (not shown in the figure). Exemplarily, the detail label generator can be the detail label generator 130 described in FIG1 .

[0263] The detail label generator is configured to binarize the third image to obtain a second ground-truth label, which is used to characterize edges in the third image. A loss calculator 1104 is configured to process the second ground-truth label and at least one feature map from the n third feature maps based on a sixth loss function to obtain a seventh loss. Loss calculator 1104 is further configured to update parameters in encoder 1101 and first decoder 1102 based on the seventh loss.

[0264] It should be understood that the above-mentioned device is used to execute part or all of the steps of the method described in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0265] Based on the method in the above embodiment, the present application provides an image processing device.

[0266] Exemplarily, FIG12 shows an image processing device. As shown in FIG12 , the image processing device 1200 may include: an encoder 1201, a first decoder 1202, and a second decoder 1203. The encoder 1201 may be used to acquire a first image, and to perform feature extraction on the first image to obtain n first feature maps, where n ≥ 2, and the scales of different first feature maps are different. The first decoder 1202 may be used to fuse the n first feature maps to obtain a first predicted label, and to output the first predicted label. The second decoder 1203 may be used to optimize the first decoder 1202 when training the first decoder 1202, wherein the scale of the feature map processed by the second decoder 1203 is larger than the scale of the feature map processed by the first decoder 1202. Exemplarily, the encoder 1201 can be the encoder 120 described in the aforementioned Figure 1, the first decoder 1202 can be the main decoding module 121 described in the aforementioned Figure 1, the second decoder 1203 can be the auxiliary decoding module 122 described in the aforementioned Figure 1, and the loss calculator 1204 can be the loss calculator 140 described in the aforementioned Figure 1.

[0267] In some embodiments, the components included in the image processing apparatus can be understood as instances (such as threads, processes, etc.) for executing a certain code. For example, the encoder 1201 can be understood as a thread or process for executing code for acquiring an image and performing feature extraction on the image.

[0268] In some embodiments, the image processing device can be deployed in a vehicle. In this case, the first image is an image captured while the vehicle is driving. The first predicted label includes one or more of the following: lane markings, guide arrows, road traffic markings, road traffic signs, crosswalks, guide strips, and poles on the road.

[0269] In some embodiments, the apparatus further includes an adversarial sample enhancer (not shown) and a loss calculator (not shown). For example, the adversarial sample enhancer may be the adversarial sample enhancer 150 described in FIG. 1 . The loss calculator may be the loss calculator 140 described in FIG. 1 .

[0270] The adversarial sample enhancer can be used to perform data enhancement on the first image to obtain a second image. The loss calculator can be used to process the first predicted label and the second predicted label using a first loss function to obtain a first loss, where the second predicted label is obtained by processing the second image using the encoder 1201 and the first decoder 1202.

[0271] In addition, the adversarial sample enhancer can also be used to add perturbations to the first image along the gradient ascent direction based on the first loss to obtain a third image. The loss calculator can also be used to process the first predicted label and the third predicted label using the first loss function to obtain a second loss, where the third predicted label is obtained by processing the third image by the encoder 1201 and the first decoder 1202; and update parameters in the encoder 1201 and the first decoder 1202 based on the second loss.

[0272] In some embodiments, the apparatus further includes an adversarial sample enhancer (not shown) and a loss calculator (not shown). For example, the adversarial sample enhancer may be the adversarial sample enhancer 150 described in FIG. 1 . The loss calculator may be the loss calculator 140 described in FIG. 1 .

[0273] The adversarial sample enhancer can be used to perform data enhancement on the first image to obtain a second image. The loss calculator can be used to process the first predicted label and the second predicted label using a first loss function to obtain a first loss, where the second predicted label is obtained by processing the second image using the encoder 1201 and the first decoder 1202.

[0274] In addition, the adversarial sample enhancer can also be used to add perturbations to the first image along the gradient ascent direction based on the first loss to obtain a third image. The loss calculator can also be used to process the first predicted label and the third predicted label using the first loss function to obtain a second loss, where the third predicted label is obtained by processing the third image by the encoder 1201 and the first decoder 1202. The loss calculator can also be used to process the first predicted label and the fourth predicted label using the second loss function to obtain a third loss, where the fourth predicted label is obtained by processing the third image by the encoder 1201 and the second decoder 1203. The loss calculator can also be used to update parameters in the encoder 1201 and the first decoder 1202 based on the second loss and the third loss.

[0275] In some embodiments, the apparatus further includes: a detail tag generator (not shown in the figure) and a loss calculator (not shown in the figure). For example, the detail tag generator may be the detail tag generator 130 described in FIG1 . The loss calculator may be the loss calculator 140 described in FIG1 .

[0276] The detail label generator may be configured to binarize the first image to obtain a true value label, where the true value label is used to characterize an edge in the first image. The loss calculator may be configured to process the true value label and at least one feature map of the n first feature maps based on a third loss function to obtain a fourth loss. The loss calculator may also be configured to update parameters in the encoder 1201 and the first decoder 1202 based on the fourth loss.

[0277] It should be understood that the above-mentioned device is used to execute the image processing method described in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0278] Based on the method in the above embodiment, an embodiment of the present application provides a terminal device. The terminal device may include: a display screen; at least one memory for storing a program; and at least one processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the above embodiment. Exemplarily, the terminal device may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, a server, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the terminal device.

[0279] Based on the methods in the above embodiments, embodiments of the present application provide a server. The server may include: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is configured to execute the methods described in the above embodiments. Exemplarily, the server may be, but is not limited to, a cloud server, a virtual machine, a hardware server, and the like. Embodiments of the present application do not impose any particular restrictions on the specific type of the server.

[0280] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0281] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0282] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0283] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0284] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0285] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: acquiring a first image; Performing feature extraction on the first image using an encoder in a neural network model to obtain n first feature maps, where n is greater than or equal to 2, and different first feature maps have different scales; fusing the n first feature maps through a first decoder in the neural network model to obtain a first predicted label, wherein the neural network model is further configured with a second decoder, the second decoder being configured to optimize the first decoder when training the neural network model, and the scale of the feature map processed by the second decoder being larger than the scale of the feature map processed by the first decoder; Output the first predicted label.

2. The method according to claim 1, characterized in that The method is performed by a vehicle, the first image is an image captured by the vehicle while it is traveling, and the first predicted label includes one or more of the following: Lane lines, guide arrows, road traffic markings, road traffic signs, pedestrian crossing lines, guide strips, and poles on the road.

3. The method according to claim 1 or 2, characterized in that The method further comprises: acquiring a second image; performing data enhancement on the second image to obtain a third image; Processing a second predicted label and a third predicted label using a first loss function to obtain a first loss, wherein the second predicted label is obtained by processing the second image using the neural network model, and the third predicted label is obtained by processing the third image using the neural network model; Based on the first loss, adding a perturbation to the second image along a gradient ascent direction to obtain a fourth image; Processing the second predicted label and the fourth predicted label using the first loss function to obtain a second loss, wherein the fourth predicted label is obtained by processing the fourth image using the neural network model; Based on the second loss, the neural network model is updated.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: acquiring a fifth image; performing a binarization process on the fifth image to obtain a first true value label, where the first true value label is used to characterize an edge in the fifth image; Processing the first true value label and at least one feature map extracted from the fifth image by the neural network model based on a second loss function to obtain a third loss; Based on the third loss, the neural network model is updated.

5. A model training method, characterized in that: The method comprises: acquiring a first image; Performing feature extraction on the first image using an encoder in a neural network model to obtain n first feature maps, where n is greater than or equal to 2, and different first feature maps have different scales; Upsampling k first feature maps of the n first feature maps by a first decoder in the neural network model to obtain k second feature maps, where k ≥ 1, wherein a scale of each feature map of the k first feature maps is larger than a scale of the remaining (nk) first feature maps of the n first feature maps; fusing the k second feature maps and the (nk) first feature maps through the first decoder to obtain a first predicted label; Fusing the k first feature maps and the feature map fused by the first decoder through a second decoder in the neural network model to obtain a second predicted label; Obtaining a loss corresponding to the first image according to the first predicted label, the second predicted label, and a sample label of the first image; The neural network model is trained based on the loss corresponding to the first image.

6. The method according to claim 5, characterized in that Obtaining a loss corresponding to the first image according to the first predicted label, the second predicted label, and the sample label of the first image specifically includes: Processing the first predicted label and the sample label of the first image based on a first loss function to obtain a first loss; Processing the second predicted label and the sample label of the first image based on a second loss function to obtain a second loss; Based on the first loss and the second loss, a loss corresponding to the first image is obtained.

7. The method according to claim 5 or 6, characterized in that The method further comprises: performing a binarization process on the first image to obtain a first true value label, where the first true value label is used to characterize an edge in the first image; Processing the first true value label and at least one feature map of the n first feature maps based on a third loss function to obtain a third loss; Based on the third loss, the neural network model is trained.

8. The method according to any one of claims 5 to 7, characterized in that: The method further comprises: Obtaining a first gradient associated with the loss corresponding to the first image based on the loss corresponding to the first image; Based on the first gradient and in a gradient ascending direction, adding a disturbance to the first image to obtain a second image; The neural network model is trained using the sample labels of the second image and the first image.

9. The method according to any one of claims 5 to 8, characterized in that: The method is executed by a cloud server, and after completing the training of the neural network model, the method further includes: Semantic segmentation is performed using the encoder and the first decoder, or semantic segmentation is performed using the encoder and the second decoder.

10. The method according to any one of claims 5 to 8, characterized in that: The method is executed by a terminal device, and after completing the training of the neural network model, the method further includes: Semantic segmentation is performed using the encoder and the first decoder.

11. The method according to any one of claims 5 to 8, characterized in that: The method is executed by a cloud server, and after completing the training of the neural network model, the method further includes: The encoder and the first decoder are deployed on a terminal device, so that the terminal device uses the encoder and the first decoder to perform semantic segmentation.

12. The method according to claim 10 or 11, characterized in that The terminal device is a vehicle.

13. A neural network model training device, characterized in that: include: an encoder, configured to acquire a first image and perform feature extraction on the first image to obtain n first feature maps, where n is greater than or equal to 2, and different first feature maps have different scales; a first decoder, configured to upsample k first feature maps of the n first feature maps to obtain k second feature maps, where k ≥ 1, wherein a scale of each feature map of the k first feature maps is larger than a scale of the remaining (nk) first feature maps of the n first feature maps; The first decoder is further configured to fuse the k second feature maps and the (nk) first feature maps to obtain a first predicted label; a second decoder, configured to fuse the k first feature maps with the feature map fused by the first decoder to obtain a second predicted label; a loss calculator, configured to obtain a loss corresponding to the first image based on the first predicted label, the second predicted label, and a sample label of the first image; The loss calculator is further used to train the encoder, the first decoder and the second decoder based on the loss corresponding to the first image.

14. The device according to claim 13, characterized in that When the loss calculator obtains the loss corresponding to the first image according to the first predicted label, the second predicted label, and the sample label of the first image, the loss calculator is specifically configured to: Processing the first predicted label and the sample label of the first image based on a first loss function to obtain a first loss; Processing the second predicted label and the sample label of the first image based on a second loss function to obtain a second loss; Based on the first loss and the second loss, a loss corresponding to the first image is obtained.

15. The device according to claim 13 or 14, characterized in that The device also includes: a detail label generator, configured to perform binarization processing on the first image to obtain a first true value label, wherein the first true value label is used to characterize an edge in the first image; The loss calculator is further configured to process the first true value label and at least one feature map of the n first feature maps based on a third loss function to obtain a third loss; The loss calculator is further configured to train the encoder, the first decoder, and the second decoder based on the third loss.

16. The device according to any one of claims 13 to 15, characterized in that: The device also includes: an adversarial example enhancer, configured to obtain a first gradient associated with the loss corresponding to the first image based on the loss corresponding to the first image; The adversarial sample enhancer is further configured to add a perturbation to the first image based on the first gradient and in a gradient ascent direction to obtain a second image; The loss calculator is further configured to train the encoder, the first decoder, and the second decoder using sample labels of the second image and the first image.

17. The device according to any one of claims 13 to 16, characterized in that: The neural network model training device is deployed in a cloud server. After the loss calculator completes training the encoder, the first decoder and the second decoder, the encoder and the first decoder are jointly used to perform semantic segmentation on the image obtained by the cloud server, or the encoder and the second decoder are jointly used to perform semantic segmentation on the image obtained by the cloud server.

18. The device according to any one of claims 13 to 16, characterized in that: The neural network model training device is deployed in the terminal device. After the loss calculator completes training the encoder, the first decoder and the second decoder, the encoder and the first decoder are used together to perform semantic segmentation on the image acquired by the terminal device.

19. The device according to any one of claims 13 to 16, characterized in that: The neural network model training device is deployed in a cloud server. After the loss calculator completes training the encoder, the first decoder and the second decoder, the encoder and the first decoder are used together to perform semantic segmentation on the image acquired by the terminal device on the terminal device.

20. The device according to claim 18 or 19, characterized in that The terminal device is a vehicle.

21. The device according to any one of claims 17 to 20, characterized in that: When the encoder and the first decoder are used together to perform semantic segmentation on the third image acquired by the cloud server or the terminal device, The encoder is configured to perform feature extraction on the third image to obtain n third feature maps, where n is greater than or equal to 2, and different third feature maps have different scales; The first decoder is used to fuse the n third feature maps to obtain a third predicted label, and output the third predicted label.

22. The device according to claim 21, characterized in that The device also includes: an adversarial sample enhancer, configured to perform data enhancement on the third image to obtain a fourth image; The loss calculator is further configured to process the third predicted label and the fourth predicted label using a fourth loss function to obtain a fourth loss, wherein the fourth predicted label is obtained by processing the fourth image using the encoder and the first decoder; The adversarial sample enhancer is further configured to add a perturbation to the third image along a gradient ascent direction based on the fourth loss to obtain a fifth image; The loss calculator is further configured to process the third predicted label and the fifth predicted label using the fourth loss function to obtain a fifth loss, where the fifth predicted label is obtained by processing the fifth image using the encoder and the first decoder; The loss calculator is further configured to update parameters in the encoder and the first decoder based on the fifth loss.

23. The device according to claim 21, characterized in that The device also includes: an adversarial sample enhancer, configured to perform data enhancement on the third image to obtain a fourth image; The loss calculator is further configured to process the third predicted label and the fourth predicted label using a fourth loss function to obtain a fourth loss, wherein the fourth predicted label is obtained by processing the fourth image using the encoder and the first decoder; The adversarial sample enhancer is further configured to add a perturbation to the third image along a gradient ascent direction based on the fourth loss to obtain a fifth image; The loss calculator is further configured to process the third predicted label and the fifth predicted label using the fourth loss function to obtain a fifth loss, where the fifth predicted label is obtained by processing the fifth image using the encoder and the first decoder; The loss calculator is further configured to process the third predicted label and the sixth predicted label using a fifth loss function to obtain a sixth loss, where the sixth predicted label is obtained by processing the fifth image using the encoder and the second decoder; The loss calculator is further configured to update parameters in the encoder and the first decoder based on the fifth loss and the sixth loss.

24. The device according to any one of claims 21 to 23, characterized in that The device also includes: a detail label generator, configured to perform binarization processing on the third image to obtain a second true value label, wherein the second true value label is used to characterize an edge in the third image; a loss calculator, configured to process the second true value label and at least one feature map of the n third feature maps based on a sixth loss function to obtain a seventh loss; The loss calculator is further configured to update parameters in the encoder and the first decoder based on the seventh loss.

25. An image processing device, characterized in that: The device comprises: an encoder configured to acquire a first image, and perform feature extraction on the first image to obtain n first feature maps, where n is greater than or equal to 2, and different first feature maps have different scales; a first decoder, configured to fuse the n first feature maps to obtain a first predicted label, and output the first predicted label; A second decoder is used to optimize the first decoder when training the first decoder, wherein the scale of the feature map processed by the second decoder is larger than the scale of the feature map processed by the first decoder.

26. The device according to claim 25, characterized in that The image processing device is deployed in a vehicle, the first image is an image collected while the vehicle is traveling, and the first predicted label includes one or more of the following: Lane lines, guide arrows, road traffic markings, road traffic signs, pedestrian crossing lines, guide strips, and poles on the road.

27. The device according to claim 25 or 26, characterized in that The device also includes: an adversarial sample enhancer, configured to perform data enhancement on the first image to obtain a second image; a loss calculator, configured to process the first predicted label and the second predicted label using a first loss function to obtain a first loss, wherein the second predicted label is obtained by processing the second image by the encoder and the first decoder; The adversarial example enhancer is further configured to add a perturbation to the first image along a gradient ascent direction based on the first loss to obtain a third image; The loss calculator is further configured to process the first predicted label and a third predicted label using the first loss function to obtain a second loss, wherein the third predicted label is obtained by processing the third image using the encoder and the first decoder; The loss calculator is further configured to update parameters in the encoder and the first decoder based on the second loss.

28. The device according to claim 25 or 26, characterized in that The device also includes: an adversarial sample enhancer, configured to perform data enhancement on the first image to obtain a second image; a loss calculator, configured to process the first predicted label and the second predicted label using a first loss function to obtain a first loss, wherein the second predicted label is obtained by processing the second image by the encoder and the first decoder; The adversarial example enhancer is further configured to add a perturbation to the first image along a gradient ascent direction based on the first loss to obtain a third image; The loss calculator is further configured to process the first predicted label and a third predicted label using the first loss function to obtain a second loss, wherein the third predicted label is obtained by processing the third image using the encoder and the first decoder; The loss calculator is further configured to process the first predicted label and the fourth predicted label using a second loss function to obtain a third loss, wherein the fourth predicted label is obtained by processing the third image using the encoder and the second decoder; The loss calculator is further configured to update parameters in the encoder and the first decoder based on the second loss and the third loss.

29. The device according to any one of claims 25 to 28, characterized in that The device also includes: a detail label generator, configured to perform binarization processing on the first image to obtain a true value label, wherein the true value label is used to characterize an edge in the first image; a loss calculator, configured to process the true value label and at least one feature map of the n first feature maps based on a third loss function to obtain a fourth loss; The loss calculator is further configured to update parameters in the encoder and the first decoder based on the fourth loss.

30. A terminal device, characterized in that: include: at least one memory for storing a program; at least one processor, configured to execute the program stored in the memory; When the program stored in the memory is executed, the processor is configured to execute the method according to any one of claims 1 to 4, or execute the method according to any one of claims 5 to 12.

31. A server, characterized in that: include: at least one memory for storing a program; at least one processor, configured to execute the program stored in the memory; When the program stored in the memory is executed, the processor is configured to execute the method according to any one of claims 1 to 4, or execute the method according to any one of claims 5 to 12.

32. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on a processor, the processor is caused to execute the method according to any one of claims 1 to 4, or the method according to any one of claims 5 to 12.

33. A computer program product, characterized in that When the computer program product runs on a processor, the processor is enabled to execute the method according to any one of claims 1 to 4, or to execute the method according to any one of claims 5 to 12.

Images