A training method, device and related equipment for image processing model

Abstract

The present invention provides a method, apparatus, and related equipment for training an image processing model. The method comprises: obtaining a training data set, the training data set including first training data and second training data, the first training data including invalid images and invalid labels matching the invalid images, wherein the invalid images are images with image defects, and the second training data including valid images, wherein the valid images include original images and standard images; inputting the training data set into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data being used to train the first network, and the second training data being used to train the second network. The method provided in an embodiment of the present invention, through the target model, better implements classification of invalid images during the image processing process, thereby resolving the problem of large amounts of resource waste in the image processing process in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a training method, apparatus and related equipment for an image processing model. Background Art

[0002] Currently, the method for judging the validity of camera-captured content is basically to identify it through the cloud. That is to say, the images from the camera's video acquisition end are processed and finally transmitted to the cloud before the video quality content can be effectively judged. However, the waste of video compression power consumption and the waste of video transmission network bandwidth are not effectively controlled. If the video content is invalid, the image processing burden will be additionally increased, resulting in a large amount of resource waste in the image processing process. Summary of the Invention

[0003] The embodiments of the present invention provide a method, apparatus and related equipment for training an image processing model, which solve the problem of a large amount of resource waste in the image processing process in the prior art.

[0004] In a first aspect, an embodiment of the present invention provides a method for training an image processing model, comprising:

[0005] Obtain a training data set, where the training data set includes first training data and second training data, where the first training data includes invalid pictures and invalid labels matching the invalid pictures, where the invalid pictures are pictures with image defects, and the second training data includes valid pictures, where the valid pictures are pictures without the image defects, and the valid pictures include original pictures and standard pictures, where the standard pictures are pictures after image enhancement is performed on the original pictures;

[0006] The training data set is input into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data is used to train the first network, the second training data is used to train the second network, the first network is used to identify whether the input target image is the invalid image, and the second network is used to enhance the image quality of the target image.

[0007] Optionally, inputting the training data set into a pre-created image recognition model for training to obtain a target model includes:

[0008] Inputting the training data set into a pre-created image recognition model for training to obtain a first image recognition model;

[0009] Performing image compensation on the training data set according to environmental parameters and shooting parameters to obtain a target training data set, wherein the environmental parameters are shooting environment parameters of the invalid image and the valid image, and the shooting parameters are camera parameters used to shoot the invalid image and the valid image;

[0010] The compensated training data set is input into the first image recognition model for training to obtain a target model.

[0011] Optionally, after inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes:

[0012] The model file of the image recognition model is converted into an Open Neural Network Exchange (ONNX) file, and the ONNX file is used to perform platform transplantation of the target model.

[0013] Optionally, before inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes:

[0014] Obtain a deep learning network model, where the deep learning network model includes the first initial network and the second initial network, where the first initial network is used to identify whether the input target image is the invalid image, and the second initial network is used to enhance the image quality of the target image:

[0015] generating model parameter correction information according to the environmental parameters and the shooting parameters;

[0016] Based on the model parameter correction information, parameters of the first initial network and the second initial network of the deep learning network model are corrected to obtain the image recognition model.

[0017] Optionally, inputting the training data set into a pre-created image recognition model for training to obtain a target model includes:

[0018] Determine a loss function in the deep learning network model, wherein the loss function includes a first loss function and a second loss function, the first loss function is based on a classification loss function and a supervised loss function of atrous pyramid pooling (ASPP), and the second loss function is based on the first loss function, a loss of a feature extraction layer, a texture feature loss, and an overall feature loss;

[0019] The deep learning network model is trained based on the loss function and the training data set to obtain a target model.

[0020] In a second aspect, an embodiment of the present invention further provides an image processing method, comprising:

[0021] Acquire a target image, where the target image is captured by a camera;

[0022] The target image is input into a target model, and a recognition result is output, wherein the target model includes a first network and a second network, the first training data is used to train the first network, and the second training data is used to train the second network, the first network is used to identify whether the input target image is an invalid image, and the invalid image is a picture with image defects, and the second network is used to enhance the image quality of the target image.

[0023] In a third aspect, the present invention further provides a training device for an image processing model, comprising:

[0024] a first acquisition module, configured to acquire a training data set, the training data set including first training data and second training data, the first training data including invalid pictures and invalid labels matching the invalid pictures, the invalid pictures being pictures with image defects, the second training data including valid pictures, the valid pictures being pictures without the image defects, the valid pictures including original pictures and standard pictures, the standard pictures being pictures after image enhancement has been performed on the original pictures;

[0025] A training module is used to input the training data set into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data is used to train the first network, the second training data is used to train the second network, the first network is used to identify whether the input target image is the invalid image, and the second network is used to enhance the image quality of the target image.

[0026] In a fourth aspect, the present invention further provides an image processing device, comprising:

[0027] A second acquisition module is used to acquire a target image, where the target image is acquired based on camera photography;

[0028] The recognition module is used to input the target image into the target model and output a recognition result, wherein the target model includes a first network and a second network, the first training data is used to train the first network, and the second training data is used to train the second network. The first network is used to identify whether the input target image is an invalid image, and the invalid image is a picture with image defects. The second network is used to enhance the image quality of the target image.

[0029] In a fifth aspect, an embodiment of the present invention further provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of the training method of the image processing model as described in any one of the first aspects are implemented, or the steps of the image processing method as described in the second aspect are implemented.

[0030] In the sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the training method of the image processing model as described in any one of the first aspects are implemented, or the steps of the image processing method as described in the second aspect are implemented.

[0031] The present invention provides a method, apparatus, and related equipment for training an image processing model. The method comprises: obtaining a training data set, wherein the training data set includes first training data and second training data, wherein the first training data includes invalid images and invalid labels matching the invalid images, wherein the invalid images are images with image defects, and the second training data includes valid images, wherein the valid images are images without the image defects, and the valid images include original images and standard images, wherein the standard images are images after image enhancement of the original images; inputting the training data set into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, wherein the first training data is used to train the first network, and the second training data is used to train the second network, wherein the first network is used to identify whether the input target image is the invalid image, and the second network is used to enhance the image quality of the target image. The method provided in the embodiment of the present invention uses the target model to better classify invalid images during the image processing process, thereby solving the problem of large-scale resource waste in the image processing process in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0033] Figure 1 Schematic diagram of the process of training an image processing model in an embodiment of the present invention;

[0034] Figure 2 Schematic diagram of the modules of the image recognition model in an embodiment of the present invention;

[0035] Figure 3 This is one of the structural diagrams of the image recognition model in an embodiment of the present invention;

[0036] Figure 4 This is the second structural diagram of the image recognition model in an embodiment of the present invention;

[0037] Figure 5 A system diagram of an image recognition model according to an embodiment of the present invention;

[0038] Figure 6 This is the third structural diagram of the image recognition model in an embodiment of the present invention;

[0039] Figure 7 Schematic diagram of the image processing method according to an embodiment of the present invention;

[0040] Figure 8 Schematic diagram of the structure of the training device of the image processing model in an embodiment of the present invention;

[0041] Figure 9 Schematic diagram of the structure of an image processing device according to an embodiment of the present invention;

[0042] Figure 10 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0044] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. A process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0045] In addition, the terms "first", "second", etc. may be used in this document to describe various directions, actions, steps or elements, but these directions, actions, steps or elements are not limited by these terms. These terms are only used to distinguish a first direction, action, step or element from another direction, action, step or element. For example, without departing from the scope of this application, a first speed difference may be referred to as a second speed difference, and similarly, a second speed difference may be referred to as a first speed difference. Both the first speed difference and the second speed difference are speed differences, but they are not the same speed difference. The terms "first", "second", etc. should not be understood to indicate or imply relative importance or to implicitly indicate the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "multiple" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0046] The present application embodiment provides a method for training an image processing model, such as Figure 1 As shown, the steps of the method include:

[0047] Step 101: Obtain a training data set, where the training data set includes first training data and second training data, where the first training data includes invalid pictures and invalid labels matching the invalid pictures, where the invalid pictures are pictures with image defects, and the second training data includes valid pictures, where the valid pictures are pictures without the image defects, and the valid pictures include original pictures and standard pictures, where the standard pictures are pictures after image enhancement is performed on the original pictures.

[0048] In this embodiment, it should be noted that the method provided by the present invention is generally applicable to cameras and other imaging devices, but can also be applied to other hardware devices, etc., and is not specifically limited in this embodiment. The first training dataset is based on the traditional PASCAL VOC 2012 dataset, and image enhancement is performed to train several pre-trained models. Multiple image enhancement algorithms are used in the pre-trained models to significantly improve the generalization ability of the network. Specifically, the first training dataset uses the PASCAL VOC dataset and undergoes data enhancement. Each image is transformed into 200 images with a linear transformation structure that includes flipping, cropping, and mirroring. Filtering transformations such as over-darkening, over-exposure, color cast, blur, banding, and low entropy are also applied to improve the generalization ability of model training. The second training dataset includes valid images, where valid images are divided into original images and standard images. The standard images are the original images after image enhancement. Optionally, invalid labels include: over-darkening, over-exposure, distorted images, banding, low entropy, color cast, occlusion, blur, etc. In other embodiments, invalid labels may also include other types, which can be adaptively adjusted and supplemented based on actual usage. This is not fully elaborated in this embodiment.

[0049] Step 102: Input the training data set into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data is used to train the first network, the second training data is used to train the second network, the first network is used to identify whether the input target image is the invalid image, and the second network is used to enhance the image quality of the target image.

[0050] In this embodiment, if Figure 2 and Figure 3 As shown, Figure 2 Schematic diagram of the modules of the image recognition model in this embodiment. Figure 3 The structure diagram of the image recognition model in this embodiment is shown in FIG. Specifically, the first network is an invalid image classification network, and the second network is an image processing network. Figure 3As shown in the figure, the grayscale parrot image uses the encoder part of the U structure to separate the feature layer images. Then, the spatial pyramid pooling layer (ASPP) operator is used to convert the five layers of feature maps into 8-bit fixed vectors. First, the five layers of feature maps are subjected to dilated convolution operations of different scales, and then upsampled into five groups of feature maps of the same size. Then, 1x1 convolution is used to increase the dimension to 8 channels. The convolution kernel is then used to perform convolution using the full-size feature map size. The purpose is to convert the 2D feature map into a 1D vector. After the final conversion into 8 feature vectors, the 8-bit vectors are fused again using the cencatenate operator and then normalized to probability values ​​through softmax, giving the probability size of 8 categories of useless images (too dark, overexposed, blurred, striped, low entropy, color cast, occlusion, and blur). Among them, the first network is network 2 in the figure, and the second network is network 1 in the figure.

[0051] It should be noted that in the prior art, the traditional u2net network only includes an image processing network, which can only perform image enhancement on images with problems. For example, it can process a blurred image into a clear image, but this image processing network cannot accurately determine where the problem with the image is. In this application, for common image problems, especially when recognizing images in the later stage, the indicators of the main 8 types of invalid images (too dark, overexposed, flowery, striped, low entropy, color cast, occlusion, and blur) are judged, so that the problem with the image can be effectively queried, thereby reversely regulating the ISP or the H264 encoder of the video, such as Figure 4 As shown, Figure 4 This is the application structure diagram of this application.

[0052] exist Figure 4 Among them, the "invented Unet classification network (identifying illegal images)" is the neural network model in this application. By judging the invalid type of the image (too dark, overexposed, flowery screen, stripes, low entropy, color cast, occlusion, blur), the ISP and video codec encoder (the module pointed by the red curved arrow in the figure, ISP: image signal processing, video codec encoder: video codecs such as H264, H265, VP8 / VP9, AV1, etc.) can be effectively adjusted.

[0053] Compared to traditional cameras, which are unable to identify invalid images that may appear in the current usage environment, this camera uploads images of invalid scenes from its own environment to form an image library. Different network weights are trained for different scenarios in the cloud, and the algorithm is distributed to the camera. This greatly improves the breadth of invalid scene identification and enhances practicality. In addition, this method is based on the Unet architecture, which is highly adaptable to different hardware platforms and consumes minimal resources. It uses different convolutional layers to collect receptive fields and uses a multi-layer receptive field fusion method to improve image recognition rate. It does not consume excessive NPU on-chip RAM during execution, greatly improving the running speed.

[0054] In this embodiment, it should be noted that the first network and the second network can be used in conjunction with each other or independently. Specifically, the first network is an invalid image classification network, and the second network is an image processing network. For example, if an image pair passes through the first network and the result is that a leaf in the image is obscured, the second network cannot obtain a complete image of the leaf no matter how it restores the image output. However, after passing through the first network, it is clear that the leaf is obscured. This information can then be uploaded to the cloud as an alarm for storage.

[0055] The present invention provides a method for training an image processing model, comprising: obtaining a training data set, the training data set including first training data and second training data, the first training data including invalid images and invalid labels matching the invalid images, the invalid images being images with image defects, the second training data including valid images, the valid images being images without the image defects, the valid images including original images and standard images, the standard images being images after image enhancement of the original images; inputting the training data set into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data being used to train the first network, the second training data being used to train the second network, the first network being used to identify whether the input target image is the invalid image, and the second network being used to enhance the image quality of the target image. The method provided in an embodiment of the present invention uses the target model to better classify invalid images during the image processing process, thereby solving the problem of a large amount of resource waste in the image processing process in the prior art.

[0056] Optionally, inputting the training data set into a pre-created image recognition model for training to obtain a target model includes:

[0057] Inputting the training data set into a pre-created image recognition model for training to obtain a first image recognition model;

[0058] Performing image compensation on the training data set according to environmental parameters and shooting parameters to obtain a target training data set, wherein the environmental parameters are shooting environment parameters of the invalid image and the valid image, and the shooting parameters are camera parameters used to shoot the invalid image and the valid image;

[0059] The compensated training data set is input into the first image recognition model for training to obtain a target model.

[0060] In this embodiment, a picture of the environment in which the camera is installed is obtained and fused with the cloud training mechanism. The enhancement method used is the same as in the above embodiment, but selective enhancement is performed according to the scene and climatic conditions. For example, in a dark indoor environment, because infrared fill light is activated, the training of black and white scenes can be ignored. Specifically, the on-site image can be collected according to the on-site environment in which the camera is installed. After the collected image is transmitted to the cloud, it is enhanced again in the cloud and trained again with the previous pre-trained model. The final trained model file is a model file with specific environment fusion information, and then the model file is sent to the NPU of the camera through the network for execution.

[0061] In this embodiment, through the training of the neural network, for some imperfect image samples, the camera is used to collect images locally, and the original model library in the cloud continues to grow and train, and then it is sent to the camera, so that the camera model is constantly improved.

[0062] Optionally, after inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes:

[0063] The model file of the image recognition model is converted into an Open Neural Network Exchange (ONNX) file, and the ONNX file is used to perform platform transplantation of the target model.

[0064] In this embodiment, the image recognition model is trained using an NV training accelerator card based on the CUDA acceleration core, such as a GTX graphics card, Tesla, VT100, or other dedicated computing cards. After the model is trained, it is converted into a general ONNX file for subsequent conversion and porting to other platforms.

[0065] It should be noted that model files are ported based on the different chips used by the camera, including heterogeneous and universal chips, and the model is converted using the corresponding platform's model compiler. After the algorithm porting file is created, the algorithm needs to be programmed to be distributed to the cloud. The camera's ability to communicate with the cloud, and feedback to the cloud on the camera's specific computing power, CPU, and NPU model, is crucial for algorithm optimization. The algorithm can be tailored to the camera's operating environment to achieve optimal inference speed for the distributed model.

[0066] For example, the network model of the present invention can identify invalid images in three scenarios. The first is to insert it into the backend of the ISP module. If the ISP's AI processing chip has the ability to dynamically deploy algorithms, this model can be used to directly send the algorithm to the internal operation of the AI-ISP. The second is to insert it into the backend of the camera video encoder, using an encoding-then-decoding method to determine whether the encoder image is normal. The third is to insert it into the backend of the cloud video decoder to analyze problems caused by network transmission and cloud decoding.

[0067] In the application of this invention, the algorithm runs in the NPU (neural network chip). After the algorithm identifies invalid image types, it can reversely control the ISP chip and video codec chip in the main chip. For example, if an image is blurry, the neural network can detect the image blur and then adjust some ISP parameters to make the image clearer. For example, if an image is distorted, the neural network will identify the problem and provide feedback to the video codec, so that the video frame will not be encoded, thereby saving encoder processing and preventing invalid work.

[0068] Optionally, before inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes:

[0069] Obtain a deep learning network model, where the deep learning network model includes the first initial network and the second initial network, where the first initial network is used to identify whether the input target image is the invalid image, and the second initial network is used to enhance the image quality of the target image:

[0070] generating model parameter correction information according to the environmental parameters and the shooting parameters;

[0071] Based on the model parameter correction information, parameters of the first initial network and the second initial network of the deep learning network model are corrected to obtain the image recognition model.

[0072] In this embodiment, after classifying images using the above network model, the internal parameters of the ISP and video encoder can be dynamically adjusted to achieve adaptive parameter adjustment capabilities. It should be noted that because coms have a very fast sampling speed, Sony's IMX series is currently the most commonly used. For example, the 1080P acquisition rate reaches 120 frames / s (4K sampling rate is 30 frames / s, 720P sampling rate is 240 frames / s). With such a fast speed, we do not need to perform invalid image classification on every frame. We can use the soft interrupts of the ISP and video encoder to perform inference classification on the image after a custom interval.

[0073] Optionally, inputting the training data set into a pre-created image recognition model for training to obtain a target model includes:

[0074] Determine a loss function in the deep learning network model, wherein the loss function includes a first loss function and a second loss function, the first loss function is based on a classification loss function and a supervised loss function of atrous pyramid pooling (ASPP), and the second loss function is based on the first loss function, a loss of a feature extraction layer, a texture feature loss, and an overall feature loss;

[0075] The deep learning network model is trained based on the loss function and the training data set to obtain a target model.

[0076] In this embodiment, see Figure 5 , Figure 5 Figure 1 is a schematic diagram of the structure of the first network, which includes the feature extraction layer (using dilated convolutions with different expansion coefficients to extract features from images of different scales, which are then fused into a feature sequence of 5 channels, which are then converted to 8 channels after a nonlinear operation of 1x1 conv, and finally converted to 8 8x1 feature vectors through fearture-size conv, which are then fused through concat and normalized with softemax to finally obtain the probability value of invalid image classification).

[0077] This algorithm is based on U 2 The net loss function is based on the loss of the encoder layer and the 8-bit 1-dimensional vector layer after ASPP downsampling, as shown in the following formula:

[0078]

[0079]

[0080] Among them, VGG is the loss of the feature extraction layer, SSIM is the texture feature loss, and MSE is the overall feature loss (PSNR can also be used instead). The loss function of this design is the Level2 layer, which includes two types of losses. The first is the classification loss function, and the second is the ASPP supervision loss function. The deep learning network model is trained through the loss function and training data set to obtain the target model.

[0081] It's important to note that the first classification loss function is the convergence loss of an 8x1 vector, used for supervised training of 8 categories. Each box element represents a probability value between 0 and 1, representing the probability of each category occurring. The weight before 0.7L_"classification" is 70%, accounting for the majority of the loss function for Network 2.

[0082] The second is the ASPP (Atrous spatial pyramid pooling) loss function, such as Figure 6 As shown in the figure, the loss between ① and ② in the figure is the ASPP (spatial pyramid pooling method) method used from ① to ②. ② is obtained after calculating ① with the void convolution kernel, and then ② is stacked up to form a pyramid feature map. This is the pooling operation of the void pyramid, and then a 5-channel feature matrix is ​​formed by fusion. Because ASPP will obtain feature maps of different scales, the loss error between ② and ① will be relatively large. Therefore, after calculating this error, its weight accounts for 30%, which becomes relatively unimportant, but it also plays a certain role in generalization and convergence.

[0083] It should be further explained that the weight parameters 0.7 and 0.3 of the loss function in Network 2 can be adaptively adjusted according to actual conditions. For example, they can be adjusted to 0.9 and 0.1 according to needs. In this way, the convergence of the model will focus more on the final result and relatively weaken the extraction process of the intermediate layer.

[0084] The present invention provides a method for training an image processing model, comprising: obtaining a training data set, the training data set including first training data and second training data, the first training data including invalid images and invalid labels matching the invalid images, the invalid images being images with image defects, the second training data including valid images, the valid images being images without the image defects, the valid images including original images and standard images, the standard images being images after image enhancement of the original images; inputting the training data set into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data being used to train the first network, the second training data being used to train the second network, the first network being used to identify whether the input target image is the invalid image, and the second network being used to enhance the image quality of the target image. The method provided in an embodiment of the present invention uses the target model to better classify invalid images during the image processing process, thereby solving the problem of a large amount of resource waste in the image processing process in the prior art.

[0085] In another embodiment, Figure 7 As shown, the present invention also provides an image processing method, comprising:

[0086] Step 401: Acquire a target image, where the target image is acquired based on camera photography.

[0087] Step 402: Input the target image into a target model and output a recognition result, wherein the target model includes a first network and a second network, the first training data is used to train the first network, and the second training data is used to train the second network, the first network is used to identify whether the input target image is an invalid image, and the invalid image is a picture with image defects, and the second network is used to enhance the image quality of the target image.

[0088] The method provided by the embodiment of the present invention uses a target model to better classify invalid images during the image processing process, thereby solving the problem of a large amount of resource waste in the image processing process in the prior art.

[0089] The embodiment of the present invention also provides a training device 500 for an image processing model, such as Figure 8 As shown, the training device 500 of the image processing model includes:

[0090] A first acquisition module 510 is configured to acquire a training data set, wherein the training data set includes first training data and second training data, wherein the first training data includes invalid images and invalid labels matching the invalid images, wherein the invalid images are images with image defects, and the second training data includes valid images, wherein the valid images are images without the image defects, and the valid images include original images and standard images, wherein the standard images are images obtained by performing image enhancement on the original images;

[0091] The training module 520 is used to input the training data set into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data is used to train the first network, and the second training data is used to train the second network, the first network is used to identify whether the input target image is the invalid image, and the second network is used to enhance the image quality of the target image.

[0092] Optionally, inputting the training data set into a pre-created image recognition model for training to obtain a target model includes:

[0093] Inputting the training data set into a pre-created image recognition model for training to obtain a first image recognition model;

[0094] Performing image compensation on the training data set according to environmental parameters and shooting parameters to obtain a target training data set, wherein the environmental parameters are shooting environment parameters of the invalid image and the valid image, and the shooting parameters are camera parameters used to shoot the invalid image and the valid image;

[0095] The compensated training data set is input into the first image recognition model for training to obtain a target model.

[0096] Optionally, after inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes:

[0097] The model file of the image recognition model is converted into an Open Neural Network Exchange (ONNX) file, and the ONNX file is used to perform platform transplantation of the target model.

[0098] Optionally, before inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes:

[0099] Obtain a deep learning network model, where the deep learning network model includes the first initial network and the second initial network, where the first initial network is used to identify whether the input target image is the invalid image, and the second initial network is used to enhance the image quality of the target image:

[0100] generating model parameter correction information according to the environmental parameters and the shooting parameters;

[0101] Based on the model parameter correction information, parameters of the first initial network and the second initial network of the deep learning network model are corrected to obtain the image recognition model.

[0102] Optionally, inputting the training data set into a pre-created image recognition model for training to obtain a target model includes:

[0103] Determine a loss function in the deep learning network model, wherein the loss function includes a first loss function and a second loss function, the first loss function is based on a classification loss function and a supervised loss function of atrous pyramid pooling (ASPP), and the second loss function is based on the first loss function, a loss of a feature extraction layer, a texture feature loss, and an overall feature loss;

[0104] The deep learning network model is trained based on the loss function and the training data set to obtain a target model.

[0105] The method provided by the embodiment of the present invention uses a target model to better classify invalid images during the image processing process, thereby solving the problem of a large amount of resource waste in the image processing process in the prior art.

[0106] The embodiment of the present invention further provides an image processing device 600, such as Figure 9 As shown, the image processing device 600 includes:

[0107] The second acquisition module 610 is used to acquire a target image, where the target image is obtained based on camera shooting;

[0108] The recognition module 620 is used to input the target image into the target model and output a recognition result, wherein the target model includes a first network and a second network, the first training data is used to train the first network, and the second training data is used to train the second network. The first network is used to identify whether the input target image is an invalid image, and the invalid image is a picture with image defects. The second network is used to enhance the image quality of the target image.

[0109] The method provided by the embodiment of the present invention uses a target model to better classify invalid images during the image processing process, thereby solving the problem of a large amount of resource waste in the image processing process in the prior art.

[0110] Figure 10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 10 As shown, the electronic device 700 includes a memory 710 and a processor 720. The number of the processor 720 in the electronic device 700 can be one or more. Figure 10 In the example, a processor 720 is used; the memory 710 and the processor 720 in the server can be connected via a bus or other means. Figure 10 The bus connection is taken as an example.

[0111] The memory 710, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the title generation method in the embodiment of the present invention. The processor 720 executes various functional applications and data processing of the server / terminal / server by running the software programs, instructions and modules stored in the memory 710, that is, realizes the above-mentioned image processing model training method or image processing method.

[0112] The processor 720 is configured to execute the computer program stored in the memory 710 to implement the following steps:

[0113] Obtain a training data set, where the training data set includes first training data and second training data, where the first training data includes invalid pictures and invalid labels matching the invalid pictures, where the invalid pictures are pictures with image defects, and the second training data includes valid pictures, where the valid pictures are pictures without the image defects, and the valid pictures include original pictures and standard pictures, where the standard pictures are pictures after image enhancement is performed on the original pictures;

[0114] The training data set is input into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data is used to train the first network, the second training data is used to train the second network, the first network is used to identify whether the input target image is the invalid image, and the second network is used to enhance the image quality of the target image.

[0115] Optionally, inputting the training data set into a pre-created image recognition model for training to obtain a target model includes:

[0116] Inputting the training data set into a pre-created image recognition model for training to obtain a first image recognition model;

[0117] Performing image compensation on the training data set according to environmental parameters and shooting parameters to obtain a target training data set, wherein the environmental parameters are shooting environment parameters of the invalid image and the valid image, and the shooting parameters are camera parameters used to shoot the invalid image and the valid image;

[0118] The compensated training data set is input into the first image recognition model for training to obtain a target model.

[0119] Optionally, after inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes:

[0120] The model file of the image recognition model is converted into an Open Neural Network Exchange (ONNX) file, and the ONNX file is used to perform platform transplantation of the target model.

[0121] Optionally, before inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes:

[0122] Obtain a deep learning network model, where the deep learning network model includes the first initial network and the second initial network, where the first initial network is used to identify whether the input target image is the invalid image, and the second initial network is used to enhance the image quality of the target image:

[0123] generating model parameter correction information according to the environmental parameters and the shooting parameters;

[0124] Based on the model parameter correction information, parameters of the first initial network and the second initial network of the deep learning network model are corrected to obtain the image recognition model.

[0125] Optionally, inputting the training data set into a pre-created image recognition model for training to obtain a target model includes:

[0126] Determine a loss function in the deep learning network model, wherein the loss function includes a first loss function and a second loss function, the first loss function is based on a classification loss function and a supervised loss function of atrous pyramid pooling (ASPP), and the second loss function is based on the first loss function, a loss of a feature extraction layer, a texture feature loss, and an overall feature loss;

[0127] The deep learning network model is trained based on the loss function and the training data set to obtain a target model.

[0128] or

[0129] Acquire a target image, where the target image is captured by a camera;

[0130] The target image is input into a target model, and a recognition result is output, wherein the target model includes a first network and a second network, the first training data is used to train the first network, and the second training data is used to train the second network, the first network is used to identify whether the input target image is an invalid image, and the invalid image is a picture with image defects, and the second network is used to enhance the image quality of the target image.

[0131] In one embodiment, an electronic device provided by an embodiment of the present invention, whose computer program is not limited to the above method operations, can also execute the training method of the image processing model provided by any embodiment of the present invention or related operations in the image processing method.

[0132] The memory 710 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal, etc. Furthermore, the memory 710 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory 710 may further include memory remotely located relative to the processor 720, and these remote memories may be connected to a server / terminal / server via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The method provided by the embodiment of the present invention uses a target model to better classify invalid images during the image processing process, thereby solving the problem of a large amount of resource waste in the image processing process in the prior art.

[0134] An embodiment of the present invention further provides a storage medium containing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform a training method for an image processing model or an image processing method, the method comprising:

[0135] Obtain a training data set, where the training data set includes first training data and second training data, where the first training data includes invalid pictures and invalid labels matching the invalid pictures, where the invalid pictures are pictures with image defects, and the second training data includes valid pictures, where the valid pictures are pictures without the image defects, and the valid pictures include original pictures and standard pictures, where the standard pictures are pictures after image enhancement is performed on the original pictures;

[0136] The training data set is input into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data is used to train the first network, the second training data is used to train the second network, the first network is used to identify whether the input target image is the invalid image, and the second network is used to enhance the image quality of the target image.

[0137] Optionally, inputting the training data set into a pre-created image recognition model for training to obtain a target model includes:

[0138] Inputting the training data set into a pre-created image recognition model for training to obtain a first image recognition model;

[0139] Performing image compensation on the training data set according to environmental parameters and shooting parameters to obtain a target training data set, wherein the environmental parameters are shooting environment parameters of the invalid image and the valid image, and the shooting parameters are camera parameters used to shoot the invalid image and the valid image;

[0140] The compensated training data set is input into the first image recognition model for training to obtain a target model.

[0141] Optionally, after inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes:

[0142] The model file of the image recognition model is converted into an Open Neural Network Exchange (ONNX) file, and the ONNX file is used to perform platform transplantation of the target model.

[0143] Optionally, before inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes:

[0144] Obtain a deep learning network model, where the deep learning network model includes the first initial network and the second initial network, where the first initial network is used to identify whether the input target image is the invalid image, and the second initial network is used to enhance the image quality of the target image:

[0145] generating model parameter correction information according to the environmental parameters and the shooting parameters;

[0146] Based on the model parameter correction information, parameters of the first initial network and the second initial network of the deep learning network model are corrected to obtain the image recognition model.

[0147] Optionally, inputting the training data set into a pre-created image recognition model for training to obtain a target model includes:

[0148] Determine a loss function in the deep learning network model, wherein the loss function includes a first loss function and a second loss function, the first loss function is based on a classification loss function and a supervised loss function of atrous pyramid pooling (ASPP), and the second loss function is based on the first loss function, a loss of a feature extraction layer, a texture feature loss, and an overall feature loss;

[0149] The deep learning network model is trained based on the loss function and the training data set to obtain a target model.

[0150] or

[0151] Acquire a target image, where the target image is captured by a camera;

[0152] The target image is input into a target model, and a recognition result is output, wherein the target model includes a first network and a second network, the first training data is used to train the first network, and the second training data is used to train the second network, the first network is used to identify whether the input target image is an invalid image, and the invalid image is a picture with image defects, and the second network is used to enhance the image quality of the target image.

[0153] Of course, the storage medium containing computer-executable instructions provided in an embodiment of the present invention is not limited to the method operations described above, and can also execute a training method for an image processing model or related operations in an image processing method provided in any embodiment of the present invention.

[0154] The method provided by the embodiment of the present invention uses a target model to better classify invalid images during the image processing process, thereby solving the problem of a large amount of resource waste in the image processing process in the prior art.

[0155] The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0156] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0157] The program code embodied on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0158] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or terminal. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0159] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A training method for an image processing model, characterized in that: include: Obtain a training data set, where the training data set includes first training data and second training data, where the first training data includes invalid pictures and invalid labels matching the invalid pictures, where the invalid pictures are pictures with image defects, and the second training data includes valid pictures, where the valid pictures are pictures without the image defects, and the valid pictures include original pictures and standard pictures, where the standard pictures are pictures after image enhancement is performed on the original pictures; Inputting the training data set into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data is used to train the first network, the second training data is used to train the second network, the first network is used to identify whether the input target image is the invalid image, and the second network is used to enhance the image quality of the target image; The image processing model runs in the neural network chip. After identifying the invalid image type, the image processing model reversely controls the ISP chip and video codec chip in the main chip.

2. The method according to claim 1, wherein Inputting the training data set into a pre-created image recognition model for training to obtain a target model comprises: Inputting the training data set into a pre-created image recognition model for training to obtain a first image recognition model; Performing image compensation on the training data set according to environmental parameters and shooting parameters to obtain a target training data set, wherein the environmental parameters are shooting environment parameters of the invalid image and the valid image, and the shooting parameters are camera parameters used to shoot the invalid image and the valid image; The compensated training data set is input into the first image recognition model for training to obtain a target model.

3. The method according to claim 1, wherein After inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes: The model file of the image recognition model is converted into an Open Neural Network Exchange (ONNX) file, and the ONNX file is used to perform platform transplantation of the target model.

4. The method according to claim 2, characterized in that Before inputting the training data set into a pre-created image recognition model for training to obtain a target model, the method further includes: Obtain a deep learning network model, where the deep learning network model includes a first initial network and a second initial network, where the first initial network is used to identify whether the input target image is the invalid image, and the second initial network is used to enhance the image quality of the target image: generating model parameter correction information according to the environmental parameters and the shooting parameters; Based on the model parameter correction information, parameters of the first initial network and the second initial network of the deep learning network model are corrected to obtain the image recognition model.

5. The method according to claim 4, characterized in that Inputting the training data set into a pre-created image recognition model for training to obtain a target model comprises: Determine a loss function in the deep learning network model, wherein the loss function includes a first loss function and a second loss function, the first loss function is based on a classification loss function and a supervised loss function of atrous pyramid pooling (ASPP), and the second loss function is based on the first loss function, a loss of a feature extraction layer, a texture feature loss, and an overall feature loss; The deep learning network model is trained based on the loss function and the training data set to obtain a target model.

6. An image processing method, characterized in that: include: Acquire a target image, where the target image is captured by a camera; Input the target image into a target model and output a recognition result, wherein the target model includes a first network and a second network, first training data is used to train the first network, and second training data is used to train the second network, the first network is used to identify whether the input target image is an invalid image, the invalid image is a picture with image defects, and the second network is used to enhance the image quality of the target image, the first training data includes invalid images and invalid labels matching the invalid images, the invalid images are pictures with image defects, the second training data includes valid images, the valid images are pictures without the image defects, the valid images include original images and standard images, and the standard images are pictures after the original images are image enhanced; The image processing method runs in a neural network chip. After identifying invalid image types, the image processing method reversely controls the ISP chip and the video codec chip in the main chip.

7. A training device for an image processing model, characterized in that: include: a first acquisition module, configured to acquire a training data set, the training data set including first training data and second training data, the first training data including invalid pictures and invalid labels matching the invalid pictures, the invalid pictures being pictures with image defects, the second training data including valid pictures, the valid pictures being pictures without the image defects, the valid pictures including original pictures and standard pictures, the standard pictures being pictures after image enhancement has been performed on the original pictures; a training module, configured to input the training data set into a pre-created image recognition model for training to obtain a target model, wherein the target model includes a first network and a second network, the first training data is used to train the first network, the second training data is used to train the second network, the first network is used to identify whether the input target image is the invalid image, and the second network is used to enhance the image quality of the target image; The image processing model runs in the neural network chip. After identifying the invalid image type, the image processing model reversely controls the ISP chip and video codec chip in the main chip.

8. An image processing device, characterized in that: include: A second acquisition module is used to acquire a target image, where the target image is acquired based on camera photography; a recognition module, configured to input the target image into a target model and output a recognition result, wherein the target model includes a first network and a second network, first training data is used to train the first network, and second training data is used to train the second network, the first network is used to identify whether the input target image is an invalid image, wherein the invalid image is a picture with image defects, and the second network is used to enhance the image quality of the target image, the first training data includes invalid images and invalid labels matching the invalid images, wherein the invalid images are pictures with image defects, the second training data includes valid images, wherein the valid images are pictures without the image defects, and the valid images include original images and standard images, wherein the standard images are pictures after image enhancement of the original images; The image processing method runs in a neural network chip. After identifying invalid image types, the image processing method reversely controls the ISP chip and the video codec chip in the main chip.

9. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of the training method of the image processing model as described in any one of claims 1 to 5 are implemented, or the steps of the image processing method as described in claim 6 are implemented.

10. A computer-readable storage medium, characterized in that The readable storage medium stores a program or instruction, which, when executed by a processor, implements the steps of the training method of the image processing model as described in any one of claims 1 to 5, or implements the steps of the image processing method as described in claim 6.

Images