An image processing method, apparatus and computer-readable storage medium

Abstract

An embodiment of the present invention discloses an image processing method, apparatus, and computer-readable storage medium. The method includes: obtaining a target image to be processed; calling an image processing model to process the target image to obtain compensation feature information corresponding to the target image, where the image processing model is trained based on reference feature information of a first sample image and compensated feature information of a second sample image, and the second sample image is generated based on the first sample image; determining second feature information corresponding to the target image according to first feature information of the target image and the compensation feature information, and the second feature information is used for classifying the target image. The embodiments of the present invention can be applied to various scenarios such as cloud technology, artificial intelligence, and intelligent transportation, and can effectively compensate the feature information of an image, improving the efficiency and accuracy of image processing.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to an image processing method, apparatus, and computer-readable storage medium. Background Art

[0002] With the continuous development and application of computer technology, people have put forward higher requirements for image processing technology, such as image recognition and classification by using image processing technology. In various life scenarios, different types of low-quality images will be generated. If these low-quality images are directly processed, it will affect the final effect of image processing.

[0003] Currently, before image processing, data enhancement operations are usually performed on image data to align low-quality images with high-quality images in the image space, and then image processing, such as image classification and image retrieval, is performed. This method is relatively complex to implement, and the processing efficiency and accuracy are both low. Summary of the Invention

[0004] Embodiments of the present invention provide an image processing method, apparatus, and computer-readable storage medium, which can effectively compensate the feature information of an image and improve the efficiency and accuracy of image processing.

[0005] In a first aspect, an embodiment of the present invention provides an image processing method, which includes: obtaining a target image to be processed; calling an image processing model to process the target image to obtain compensation feature information corresponding to the target image, where the image processing model is trained based on the reference feature information of a first sample image and the compensated feature information of a second sample image, and the second sample image is generated based on the first sample image; determining second feature information corresponding to the target image according to the first feature information and the compensation feature information of the target image, where the second feature information is used for classifying the target image.

[0006] In a second aspect, an embodiment of the present invention provides an image processing apparatus, which includes:

[0007] An obtaining module, configured to obtain a target image to be processed;

[0008] A processing module, configured to call an image processing model to process the target image to obtain compensation feature information corresponding to the target image, where the image processing model is trained based on the reference feature information of a first sample image and the compensated feature information of a second sample image, and the second sample image is generated based on the first sample image;

[0009] A determining module, configured to determine second feature information corresponding to the target image according to the first feature information and the compensation feature information of the target image, where the second feature information is used for classifying the target image.

[0010] In a third aspect, an embodiment of the present invention provides a server, which includes a processor, a network interface, and a storage device. The processor, the network interface, and the storage device are interconnected. Among them, the network interface is controlled by the processor to transmit and receive data, the storage device is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the above-mentioned image processing method.

[0011] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions, and the program instructions are executed by a processor to execute the above-mentioned image processing method.

[0012] In a fifth aspect, an embodiment of the present application provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned image processing method.

[0013] In the embodiment of the present invention, by obtaining a target image to be processed, calling an image processing model to process the target image, obtaining compensation feature information corresponding to the target image, and determining second feature information corresponding to the target image according to the first feature information and the compensation feature information of the target image, alignment in the deep feature space with a corresponding high-quality image is achieved. The second feature information can be used to perform image classification, restoration, etc. on the target image, thereby simplifying the image processing process and improving the image processing efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0015] Figure 1 is a schematic diagram of the architecture of an image processing system provided by an exemplary embodiment of the present application;

[0016] Figure 2 is a schematic diagram of the process of an image processing provided by an exemplary embodiment of the present application;

[0017] Figure 3 is a schematic diagram of the process of an image processing method provided by an exemplary embodiment of the present application;

[0018] Figure 4 is a schematic flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0019] Figure 5 is a schematic flowchart of a high-quality image screening provided by an exemplary embodiment of the present application;

[0020] Figure 6 is a schematic framework diagram of a model training provided by an exemplary embodiment of the present application;

[0021] Figure 7 is a schematic structural diagram of a feature repair network provided by an exemplary embodiment of the present application;

[0022] Figure 8 is a schematic block diagram of an image processing apparatus provided by an exemplary embodiment of the present application;

[0023] Figure 9 is a schematic block diagram of a server provided by an exemplary embodiment of the present application. Detailed implementation manners

[0024] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0025] Please refer to Figure 1 , Figure 1 is a schematic architecture diagram of an image processing system provided by an exemplary embodiment of the present application. As Figure 1 shown, the image processing system may specifically include a terminal device 101 and a server 102, and the terminal device 101 and the server 102 are connected through a network, for example, through a wireless network connection or the like.

[0026] The terminal device 101 is also referred to as a terminal, a user equipment (UE), an access terminal, a user unit, a mobile device, a user terminal, a wireless communication device, a user agent, or a user device. The terminal device may be a smart TV, a handheld device with wireless communication capabilities (such as a smart phone, a tablet computer), a computing device (such as a personal computer (PC), a vehicle-mounted device, a wearable device, or other intelligent devices), but is not limited thereto.

[0027] Server 102 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0028] In one embodiment, as Figure 2 shown, server 102 can call the feature extraction network included in the image processing model to perform feature extraction processing on the input image to be processed to obtain the first feature information of the image to be processed, and call the feature repair network included in the image processing model to perform feature repair processing on the original feature information and the first feature information of the input image to be processed to obtain the second feature information of the image to be processed, realizing information alignment with the corresponding high-quality image in the deep feature space, and performing image processing operations according to the second feature information after optimized processing of the image to be processed. This method simplifies the image processing process and improves the image processing efficiency and accuracy.

[0029] In one embodiment, server 102 can use the picture as the data to be processed, obtain the second feature information of the image to be processed by inputting the picture to be processed on the terminal device 101, and perform image classification operations, image retrieval operations, etc. using the second feature information; similarly, server 102 can also use the video as the data to be processed, obtain the image frames that need to be processed in the video, and then perform subsequent image processing operations, and so on.

[0030] It can be understood that the schematic diagram of the system architecture described in the embodiments of the present application is for more clearly explaining the technical solutions of the embodiments of the present application, and does not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those of ordinary skill in the art know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0031] As Figure 3 shown, Figure 3 is a schematic flowchart of an image processing method provided by an exemplary embodiment of the present application. Taking the method applied to Figure 1 server 102 in it as an example for illustration, the method may include the following steps:

[0032] S301. Obtain the target image to be processed.

[0033] Among them, the target image to be processed is an image that requires image processing. The target image referred to here can be either picture data, an image frame in a video, or an image generated by computer technologies such as AI algorithms, etc. This application does not limit the type of the target image to be processed.

[0034] In one embodiment, the target image to be processed can be a relatively blurred and low-quality image. Similarly, a clear image can also be used as the target image to be processed, and the goal to be achieved in this embodiment can also be realized. If the target image to be processed is a relatively blurred and low-quality image, then the second feature information corresponding to the low-quality image obtained by this embodiment (that is, the feature information of the clear image corresponding to the low-quality image) can be used to well perform subsequent image processing. Compared with directly using the low-quality image for image processing, its processing efficiency and accuracy are greatly improved; if the target image to be processed is a relatively clear and high-quality image, then using the second feature information corresponding to the relatively clear and high-quality image obtained by this embodiment for subsequent image processing, the improvement effect is limited, but it does not affect the ultimate goal of this embodiment.

[0035] In one embodiment, the target image to be processed can be a clear image or a low-quality image. However, if there is a large amount of high-quality data and only a small part of it is low-quality data, it is not very valuable to process these large amounts of high-quality data, and the excessive high-quality data will also increase the burden on the execution of the entire system. Therefore, the target image can be screened through image processing methods to filter out high-quality images. Only when the target image is a low-quality image, the feature information needs to be repaired, that is, the subsequent steps S302 to S303 are executed, thereby improving the efficiency of image processing. The specific image screening methods can include the following two: The first method is to use the Sobel Operator as the basis for judging image quality. The Sobel Operator is used for edge detection of images, and it estimates the magnitude of the gradient for each pixel. Pixels with a larger gradient value indicate that there is a large color difference between it and the surrounding pixels, so this pixel must be located on the edge of the image. On the contrary, pixels with a smaller gradient mean that their colors are similar to those of adjacent pixels, that is, the pixel is not on the edge of the image. It should be noted that the Sobel Operator does not return a binary result indicating whether the pixel is on the edge of the image, but a gray value in the range of [0.0, 1.0] representing the "steepness" of the edge: a value of 0 means very flat and no color difference from the surrounding pixels; a value of 1 means very steep and a large color difference from the surrounding pixels. Usually, the Sobel inverse image (1 - c) is often more intuitive and effective. At this time, white represents flat and not on the edge of the image, while black represents steep and on the edge of the image. Calculate the clarity of the image through the average value of the Sobel Operator. The lower the average value, the more blurred the image is determined to be, and the higher the average value, the clearer the image is determined to be. Relative clear images can be screened through a manually set Sobel threshold. The second method is to convert the RBG image into the Lab space and calculate the average value. The lower the average value, the darker the image is. For example, an image with an average value less than or equal to a certain threshold can be regarded as a low-quality image; the higher the average value, the brighter the image is. For example, an image with an average value greater than or equal to a certain threshold can be regarded as a high-quality image. Therefore, the target image that needs feature repair should be an image with a lower average value. Using this rule, the image quality can also be judged and screened.

[0036] S302. Call an image processing model to process the target image to obtain compensation feature information corresponding to the target image. The image processing model is trained based on the reference feature information of the first sample image and the compensated feature information of the second sample image, and the second sample image is generated based on the first sample image.

[0037] Since the image information of each image can be obtained through image processing, the image information here can be the parameter information of the image such as image exposure, image grayscale, image size, etc., or the feature information that can be used to identify the image features such as image edge features, image texture features, image spatial relationship features, etc. Therefore, this application uses an image processing module to perform image processing on the image, and obtains the feature information of the clear image aligned with the features of the low-quality image in the high-dimensional space through it for subsequent image processing operations.

[0038] Specifically, the image processing module of this application includes two network models, namely a feature extraction network and a feature repair network. The main function of the feature extraction network is to perform feature extraction processing on the target image to obtain the high-dimensional feature information of the target image, which serves as the basis for subsequent feature repair using the high-dimensional feature information of the image; the main function of the feature repair network is to perform feature repair processing on the original feature information and high-dimensional feature information of the target image to obtain the compensated feature information of the target image. Using this compensated feature information and the original feature information of the target image, the feature information of the clear image aligned with the features of the target image in the high-dimensional space can be obtained, that is, the second feature information mentioned above. The output of the image processing module of this application can be to directly output the compensated feature information for subsequent processing, or to build in a feature operation algorithm to output the second feature information through feature operation.

[0039] In one embodiment, the image processing module processes the target image to obtain the compensated feature information of the target image, and superimposes the original features and compensated feature information of the target image in a coefficient ratio of 1:1. Finally, the output result of the image processing module is the feature information after the target image is repaired, and image classification or image restoration is performed according to the repaired feature information.

[0040] If the image processing model wants to achieve good processing results, the model training stage is essential. Through training, the image processing model can better repair the image feature information, so as to achieve the effect of processing the image using the feature information of the image.

[0041] In one embodiment, the image processing model is trained based on the reference feature information of the first sample image and the compensated feature information of the second sample image, and the second sample image is generated based on the first sample image. The first sample image and the second sample image here have a corresponding relationship. Specifically, the second sample image is obtained by performing image damage processing on the first sample image. The purpose of doing this is to use the first sample image as a reference image, and naturally the feature information of the first sample image is also reference feature information. Repairing the feature information of the damaged image generated from the reference image and then combining it with the feature information of the reference image for training the image processing model can obtain better training effects.

[0042] S303. Determine the second feature information corresponding to the target image according to the first feature information and the compensated feature information of the target image, where the second feature information is used to perform classification processing on the target image.

[0043] Specifically, the first feature information here includes the original feature information of the target image. Perform feature calculation on the original feature information and the compensated feature information of the target image to obtain the second feature information of the target image. The second feature information here is the repaired high-dimensional feature information corresponding to the target image, and the second feature information can be used for image processing of the target image.

[0044] In one embodiment, the original feature information, the first feature information, the second feature information, and the compensated feature information are all essentially feature vectors, and feature vectors can be easily used for feature vector calculation. Perform arithmetic addition on the feature vectors of the original feature information and the compensated feature information of the obtained target image to obtain the feature vector of the second feature information of the target image, and finally use the feature vector of the second feature information to perform image classification on the target image.

[0045] In the embodiment of the present application, first obtain the target image to be processed, then call the image processing model to process the target image to obtain the compensated feature information corresponding to the target image. The image processing model is trained based on the reference feature information of the first sample image and the compensated feature information of the second sample image, and the second sample image is generated based on the first sample image. Finally, determine the second feature information corresponding to the target image according to the first feature information and the compensated feature information of the target image, and perform classification processing on the target image according to the second feature information, shortening the technical path of image processing and improving the processing efficiency.

[0046] As Figure 4 shown, Figure 4 is a schematic flowchart of an image processing method provided by another exemplary embodiment of the present application. Taking this method as applied to Figure 1Taking the server 102 in [as an example], the method may include the following steps:

[0047] S401. Obtain a plurality of sample image pairs, where each sample image pair in the plurality of sample image pairs includes a first sample image and a second sample image.

[0048] Specifically, the method provided in this embodiment needs to train an image processing model so as to meet the requirement of outputting compensation feature information corresponding to a target image. Therefore, high-quality images and low-quality images corresponding to the high-quality images can be used to train the image processing model. The plurality of sample image pairs referred to here include multiple groups of first sample images and second sample images with a corresponding relationship. The first sample image is a high-quality image used for model training, and the second sample image is a low-quality image used for model training. To make the model training more efficient and accurate, the second sample image we use is obtained by damaging the first sample image. Generally speaking, we use the high-quality first sample image as a reference sample. By comparing the generated low-quality second sample image with the reference sample, it is more intuitive and accurate. Similarly, taking the feature information of the high-quality first sample image as a reference feature information, by comparing the feature information of the generated low-quality second sample image with the reference feature information, it is also intuitive and accurate.

[0049] In one embodiment, after obtaining the first sample image, an image damaging operation can be performed on the first sample image to obtain the second sample image. The image damaging processing referred to here includes but is not limited to motion blur, contrast adjustment, brightness adjustment, adding scratches, image fogging, and adding noise. A single damaging means can be used to perform the damaging processing on the first sample image to obtain a second sample image with a relatively small degree of damage, or a combination of multiple damaging means can be used. Of course, using a combination of multiple damaging means can obtain a second sample image with a relatively large degree of damage. The average degree of damage of the second sample image will affect the training efficiency of the image processing model to a certain extent. In addition to the above image damaging means, other technical means capable of performing image damaging processing should also fall within the protection scope of this embodiment. After obtaining the second sample image obtained by damaging the first sample image, each first sample image and its corresponding second sample image are used as a sample image pair, and finally a plurality of sample image pairs are obtained for subsequent training of the image processing model.

[0050] In one embodiment, before obtaining the second sample image corresponding to the above first sample image, it is also necessary to first obtain the first sample image. The first sample image is a high-quality image used for model training, and the second sample image is a low-quality image used for model training, so as to make the model training more efficient and accurate. Therefore, a batch of high-quality images can be screened as the first sample images. The method for obtaining high-quality images is as follows Figure 5 As shown, the first type of method is to obtain the brightness score or clarity score of the original image through image processing of the original image, and then determine whether to select this image as a high-quality image by determining whether it meets the high-quality image threshold range; the second type of method is to perform feature extraction on the original image and the damaged image corresponding to the original image through a feature extraction model to obtain the first feature information of the original image and the first feature information of the damaged image, and then determine whether to select this image as a high-quality image by determining whether the feature difference between the first feature information of the original image and the first feature information of the damaged image meets the high-quality image threshold range.

[0051] The method for obtaining the first sample image includes the following steps:

[0052] (1) Obtain an original image set, which includes a plurality of original images.

[0053] The original data of the original data set here can include various types of image data, which can be image data obtained from the Internet through data capture, the photo album stored in a mobile device, or even application screenshot data generated by software products. All of these can be used as the original data for subsequent screening of high-quality image data.

[0054] (2) Obtain the quality evaluation parameters of each original image in the plurality of original images. The quality evaluation parameters include one or more of the degree of blurriness, the degree of dimness, and the average difference of image features. The average difference of image features is the average difference of image features between the original image and the corresponding damaged image.

[0055] Specifically, the calculation of the blur degree can use the Sobel operator as the basis for judging the image quality. The clarity of the image is calculated through the average value of the Sobel operator. The lower the average value, the more blurred the image is determined to be, and the higher the average value, the clearer the image is determined to be. By setting a Sobel threshold artificially, relatively clear images can be screened out. The average value of the Sobel operator calculated by this method is the quality evaluation parameter of this method; it is also possible to convert the RBG image into the Lab space and calculate the average value of the image vector. The lower the average value, the darker the image is, and the higher the average value, the brighter the image is. Using this rule, the image quality can also be judged and screened. The average value of the image vector calculated by this method is the quality evaluation parameter of this method; similarly, the average difference between the first feature information of each original image and the first feature information of the corresponding damaged image can also be used as the quality evaluation parameter to judge and screen the image quality.

[0056] Specifically, the quality evaluation parameters of each original image in multiple original images can be obtained through the following method: First, call the feature extraction network included in the image processing model to obtain the first feature information of each original image in multiple original images and the first feature information of the damaged images obtained by subjecting each original image to damage processing; then, according to the first feature information of each original image and the first feature information of the corresponding damaged image, determine the average difference of the image features between each original image and the corresponding damaged image; finally, use the average difference of the corresponding image features as the quality evaluation parameter of each original image.

[0057] (3) Use the original images among the multiple original images whose quality evaluation parameters meet the image quality screening conditions as the first sample images.

[0058] After obtaining the quality evaluation parameters of each original image, compare the quality evaluation parameters of each original image with the quality screening conditions, and finally use the original images that meet the quality screening conditions as the first sample images.

[0059] In one embodiment, the Sobel threshold characterizing the blur degree of a high-quality image can be set empirically, and then the average value of the Sobel operator of the original image is calculated and compared with the Sobel threshold characterizing the blur degree of the high-quality image. If the average value of the Sobel operator of the original image is greater than or equal to the Sobel threshold characterizing the blur degree of the high-quality image, the original image is determined as the first sample image. Also, the image vector threshold characterizing the dimness degree of the high-quality image can be set empirically, and then the average value of the image vector of the original image in the Lab color space is calculated and compared with the image vector threshold characterizing the dimness degree of the high-quality image. If the average value of the image vector of the original image in the Lab color space is greater than or equal to the image vector threshold characterizing the dimness degree of the high-quality image, the original image is determined as the first sample image; alternatively, the feature vector threshold characterizing the image feature difference of the high-quality image can be set empirically, and then the difference between the first feature information of the original image and the first feature information of the damaged image corresponding to the original image is calculated and compared. If the difference between the first feature information of the original image and the first feature information of the damaged image corresponding to the original image is greater than or equal to the feature vector threshold characterizing the image feature difference of the high-quality image, the original image is determined as the first sample image. The method for determining the original image as the first sample image is not limited to the above three methods, and it can also be a combination of one or two or three of them, performing multi-dimensional high-quality image screening on the original image. For example, the original image with the average value of the Sobel operator of the original image greater than or equal to the Sobel threshold and the difference between the first feature information of the original image and the first feature information of the damaged image corresponding to the original image greater than or equal to the feature vector threshold is used as the first sample image, where the Sobel threshold is used to characterize the blur degree of the high-quality image, and the feature vector threshold is used to characterize the image feature difference of the high-quality image.

[0060] S402. Train the initial model using the first feature information of the first sample image and the second feature information corresponding to the second sample image to obtain an image processing model, where the second feature information is the compensated feature information.

[0061] Specifically, the training process of training the initial model using the first feature information of the first sample image and the second feature information corresponding to the second sample image is as Figure 6 shown. The clear image in the figure represents the first sample image, the inferior image in the figure represents the second sample image, and the first feature information in the figure includes high-dimensional feature information. The specific method for training the initial model using the first feature information of the first sample image and the second feature information corresponding to the second sample image includes the following steps:

[0062] (1) Call the feature extraction network included in the initial model to perform feature extraction processing on the first sample image and the second sample image respectively, to obtain the first feature information of the first sample image and the first feature information of the second sample image.

[0063] The main function of the feature extraction network is to perform feature extraction processing on the target image to obtain the high-dimensional feature information of the target image, which serves as the basis for subsequent feature restoration using the high-dimensional feature information of the image. An image is essentially a multi-dimensional matrix, so corresponding calculations and processing can be performed on the image. Matrix operations are common and mature in both mathematics and computers. Converting the operations on the image into operations on the matrix is the method used by all image processing tools for image processing. For example, a grayscale image is a two-dimensional matrix, and each point in the matrix is a grayscale value. A color image is also a matrix, but each point in the matrix is not a single value, but an array containing 3 values, which are the RGB values.

[0064] From a macroscopic perspective, the feature extraction network consists of multiple convolutional layers. Each convolutional layer contains one or more convolutional kernels. These convolutional kernels scan the entire image from left to right and from top to bottom in sequence to obtain the output data called the feature map. The convolutional layers at the front of the network capture local and detailed information of the image and have small receptive fields, that is, each pixel of the output image only uses a very small range of the input image. The receptive fields of the subsequent convolutional layers increase layer by layer, which are used to capture more complex and abstract information of the image. After the operations of multiple convolutional layers, the abstract representations of the image at various different scales are finally obtained, and the image feature vector obtained through the operations can be used as the feature information of the image.

[0065] Before calling the feature extraction network included in the initial model to perform feature extraction processing on the first sample image and the second sample image in step 402, the feature extraction network also needs to be trained. To simplify the operation of training the initial model, the first sample image can be used as the training sample of the feature extraction network without having to obtain high-quality training samples again. The implementation method of training the feature extraction network needs to first obtain multiple first sample images and label reference category labels for each first sample image, then select a loss function and set the initial parameters of the loss function, then input the first sample image into the feature extraction network to obtain the predicted category label, and finally use the loss function to calculate the predicted category label and the reference category label of the first sample image, and adjust the network parameters of the feature extraction network until the preset accuracy value is met, and finally obtain the trained feature extraction network.

[0066] (2) Calling the feature restoration network included in the initial model to perform feature restoration processing on the original feature information of the second sample image and the first feature information to obtain compensated feature information corresponding to the second sample image.

[0067] The main function of the feature restoration network is to perform feature restoration processing on the original feature information and high-dimensional feature information of the image to obtain the compensated feature information of the image. In theory, various different model structures can be used to achieve the purpose of outputting the compensated feature information of the image. In general, the input of the feature restoration module is the original feature information and high-dimensional feature information of the image. After convolution processing of multiple different convolutional layers, different feature information corresponding to different convolutional layers of the image is obtained. Finally, the features output by each layer are sent to a convolutional layer for integrating feature information. The final output result of the feature restoration network is the compensated feature information used to perform feature restoration processing on the original image.

[0068] In one embodiment, the feature repair network can use Figure 7 The network structure shown in the figure, the feature repair network in this embodiment is composed of an input layer, a convolution layer, an activation layer, a pooling layer, a connection layer and an output layer. The convolution layer is composed of three blocks for feature extraction, each of which contains two 3x3 convolution layers, which can perform multi-dimensional feature extraction; the pooling layer is used to reduce the image size and improve the training speed. Although some image information is lost in this way, it also increases robustness. In this embodiment, three forms of general pooling, overlapping pooling and pyramid pooling can be used; the connection layer is used to integrate the feature convolution layer, which is composed of a 1x1 convolution layer, and the purpose of feature integration is achieved by performing feature vector operations on it. Another problem that needs attention is that there are two problems when convolving the original image directly. First, the image will be reduced after each convolution; second, compared with the points in the middle of the picture, the points on the edge of the picture are calculated very few times in the convolution, which makes the edge information easy to lose. If you want the output of the feature restoration network to be compensated feature information of the same size as the original features of the input image, you can use the padding method. Before each convolution, fill a circle of blank space around the image so that the image after convolution is the same size as the original one. At the same time, the original edges are also calculated more times, and finally you can get compensated feature information of the same size as the original features of the input image.

[0069] (3) Determine second feature information corresponding to the second sample image according to the first feature information of the second sample image and the corresponding compensation feature information.

[0070] Specifically, the first feature information here includes the original feature information of the target image. The original feature information of the target image and the compensation feature information are subjected to feature calculation to obtain the second feature information of the target image. Here, the second feature information is the restored high-dimensional feature information corresponding to the target image. Among them, for the specific implementation of determining the second feature information corresponding to the second sample image according to the first feature information and the corresponding compensation feature information of the second sample image, refer to the relevant description of step S303 in the foregoing embodiments, which will not be elaborated here.

[0071] (4) Adjust the network parameters of the initial model according to the difference between the first feature information of the first sample image and the second feature information corresponding to the second sample image to obtain an image processing model, where the network parameters include the network parameters of one or both of the feature extraction network and the feature repair network.

[0072] Specifically, the network parameters of the initial model need to be set in advance. The network parameters of the initial model can be set by a random generation method, or can be set in advance according to existing experience. Of course, setting parameters according to existing experience can reduce the amount of training data to a certain extent and is more suitable for the case of fewer training image samples, but ultimately can achieve the ultimate goal of completing model training. After setting the network parameters of the initial model, use the difference between the first feature information of the first sample image and the second feature information corresponding to the second sample image to train the initial model, and adjust the network parameters of the initial model until the difference between the first feature information of the first sample image and the second feature information corresponding to the second sample image meets the preset error tolerance range, and finally obtain a trained image processing model.

[0073] In one embodiment, the loss function can adopt the mean square error loss function. Adjusting the network parameters of the initial model using the mean square error loss function includes the following steps:

[0074] 1) Initialize the parameters of the forward calculation formula with random values;

[0075] 2) Substitute the samples and calculate the predicted values of the output;

[0076] 3) Use the loss function to calculate the error between the predicted value and the label value (true value);

[0077] 4) According to the derivative of the loss function, backpropagate the error along the direction of the minimum gradient to correct each weight value in the forward calculation formula;

[0078] 5) Return to step 2 and execute the subsequent steps until the value of the loss function reaches a preset value and stop the iteration.

[0079] In one embodiment, the initial model is trained by a mean squared error loss function to make the first feature information of the first sample image as close as possible to the second feature information corresponding to the second sample image. During the training process, the previously trained feature extraction network can be directly used, and the network parameters of the feature extraction network are frozen unchanged during the subsequent training of the feature repair network in the initial model, and finally a trained image processing model is obtained; alternatively, a joint training method can be used. When training the initial model using the mean squared error loss function, the network parameters in both the feature extraction network and the feature repair network are adjusted simultaneously, and finally a trained image processing model is obtained. This embodiment does not limit the above training methods. In actual situations, according to the comparison of the advantages and disadvantages of the training results of each training method, actual adjustments are made to make the first feature information of the first sample image as close as possible to the second feature information corresponding to the second sample image.

[0080] S403. Obtain a target image to be processed.

[0081] For the specific implementation manner of step S403, refer to the relevant description of step S301 in the foregoing embodiment, and details are not described herein again.

[0082] S404. Invoke the image processing model to process the target image to obtain compensation feature information corresponding to the target image. The image processing model is trained based on the reference feature information of the first sample image and the compensated feature information of the second sample image, and the second sample image is generated based on the first sample image.

[0083] For the specific implementation manner of step S404, refer to the relevant description of step S302 in the foregoing embodiment, and details are not described herein again.

[0084] S405. Determine the second feature information corresponding to the target image according to the first feature information and the compensation feature information of the target image. The second feature information is used for classifying the target image.

[0085] For the specific implementation manner of step S405, refer to the relevant description of step S303 in the foregoing embodiment, and details are not described herein again.

[0086] In the embodiments of the present application, first, a plurality of sample image pairs are obtained, and each sample image pair in the plurality of sample image pairs includes a first sample image and a second sample image; secondly, an initial model is trained using the first feature information of the first sample image and the second feature information corresponding to the second sample image to obtain an image processing model, where the second feature information is compensated feature information; then, a target image to be processed is obtained; next, the image processing model is called to process the target image to obtain the compensated feature information corresponding to the target image, and the image processing model is trained based on the reference feature information of the first sample image and the compensated feature information of the second sample image, and the second sample image is generated based on the first sample image; finally, the second feature information corresponding to the target image is determined according to the first feature information and the compensated feature information of the target image, and the target image is classified according to the second feature information. In this embodiment, the first sample image and the second sample image are used to train the feature extraction network and the feature repair network in the image processing model, standardizing the model training process, improving the accuracy of model prediction, and improving the utilization rate of training samples. Overall, this method shortens the technical path of image processing and improves the processing efficiency and accuracy.

[0087] See Figure 8 , which is a schematic block diagram of an image processing device provided by an embodiment of the present invention. Among them, the image processing device may specifically include:

[0088] An acquisition module 801, configured to acquire a target image to be processed;

[0089] A processing module 802, configured to call an image processing model to process the target image to obtain the compensated feature information corresponding to the target image, where the image processing model is trained based on the reference feature information of the first sample image and the compensated feature information of the second sample image, and the second sample image is generated based on the first sample image;

[0090] A determination module 803, configured to determine the second feature information corresponding to the target image according to the first feature information and the compensated feature information of the target image, where the second feature information is used to classify the target image.

[0091] Optionally, the processing module 802 is specifically configured to:

[0092] Call the feature extraction network included in the image processing model to perform feature extraction processing on the target image to obtain the first feature information of the target image, where the first feature information includes high-dimensional feature information;

[0093] Call the feature repair network included in the image processing model to perform feature repair processing on the original feature information and the first feature information of the target image, and obtain the compensated feature information corresponding to the target image.

[0094] Optionally, the obtaining module 801 is further configured to:

[0095] Obtain a plurality of sample image pairs, where each sample image pair in the plurality of sample image pairs includes a first sample image and a second sample image.

[0096] Optionally, the processing module 802 is further configured to:

[0097] Use the first feature information of the first sample image and the second feature information corresponding to the second sample image to train the initial model, and obtain an image processing model, where the second feature information is the compensated feature information.

[0098] Optionally, the processing module 802 is specifically configured to:

[0099] Call the feature extraction network included in the initial model to perform feature extraction processing on the first sample image and the second sample image respectively, and obtain the first feature information of the first sample image and the first feature information of the second sample image.

[0100] Optionally, the processing module 802 is specifically configured to:

[0101] Call the feature repair network included in the initial model to perform feature repair processing on the original feature information and the first feature information of the second sample image, and obtain the compensated feature information corresponding to the second sample image.

[0102] Optionally, the determining module 803 is specifically configured to:

[0103] Determine the second feature information corresponding to the second sample image according to the first feature information of the second sample image and the corresponding compensated feature information.

[0104] Optionally, the processing module 802 is specifically configured to:

[0105] Adjust the network parameters of the initial model according to the difference between the first feature information of the first sample image and the second feature information corresponding to the second sample image, and obtain an image processing model, where the network parameters include the network parameters of one or both of the feature extraction network and the feature repair network.

[0106] Optionally, the processing module 802 is further configured to:

[0107] For each of the multiple first sample images, perform image damage processing to obtain a second sample image corresponding to each of the first sample images, where the image damage processing includes one or more of dynamic blur, contrast adjustment, brightness adjustment, adding scratches, image fogging, and adding noise;

[0108] Determine a plurality of sample image pairs according to each of the first sample images and the corresponding second sample images.

[0109] Optionally, the obtaining module 801 is further configured to:

[0110] Obtain an original image set, where the original image set includes a plurality of original images;

[0111] Obtain a quality evaluation parameter for each of the plurality of original images, where the quality evaluation parameter includes one or more of a blur degree, a dimness degree, and an average difference of image features, and the average difference of image features is the average difference of image features between the original image and the corresponding damaged image;

[0112] Use the original images among the plurality of original images whose quality evaluation parameters meet the image quality screening conditions as the first sample images.

[0113] Optionally, the processing module 802 is specifically configured to:

[0114] Call the feature extraction network included in the image processing model to obtain first feature information of each of the plurality of original images and first feature information of the damaged image corresponding to each of the original images;

[0115] Determine the average difference of image features between each of the original images and the corresponding damaged image according to the first feature information of each of the original images and the first feature information of the corresponding damaged image;

[0116] Use the corresponding average difference of image features as the quality evaluation parameter for each of the original images.

[0117] Optionally, the processing module 802 is further configured to:

[0118] Use a plurality of first sample images and the reference class labels of each first sample image to pre-train the feature extraction network included in the initial model.

[0119] It should be noted that the functions of the functional modules of the image processing device in the embodiments of the present application can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can refer to the relevant descriptions of the above method embodiments, which will not be elaborated here.

[0120] See Figure 9, which is a schematic block diagram of a server provided by an embodiment of the present invention. The server in this embodiment shown in the figure may include: a processor 901, a storage device 902, and a network interface 903. Data interaction can be performed among the processor 901, the storage device 902, and the network interface 903.

[0121] The storage device 902 may include a volatile memory, such as a random-access memory (RAM); the storage device 902 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the storage device 902 may further include a combination of the above types of memories.

[0122] The processor 901 may be a central processing unit (CPU). In one embodiment, the processor 901 may also be a Graphics Processing Unit (GPU). The processor 901 may also be a combination of a CPU and a GPU. In one embodiment, the storage device 902 is used to store program instructions, and the processor 901 may call the program instructions to perform the following operations:

[0123] Obtain a target image to be processed;

[0124] Call an image processing model to process the target image to obtain compensation feature information corresponding to the target image. The image processing model is trained based on the reference feature information of a first sample image and the feature information after compensation of a second sample image, and the second sample image is generated based on the first sample image;

[0125] Determine second feature information corresponding to the target image according to the first feature information of the target image and the compensation feature information. The second feature information is used for classifying the target image.

[0126] Optionally, the processor 901 is specifically used for:

[0127] Call a feature extraction network included in the image processing model to perform feature extraction processing on the target image to obtain first feature information of the target image. The first feature information includes high-dimensional feature information;

[0128] Invoking the feature repair network included in the image processing model to perform feature repair processing on the original feature information and the first feature information of the target image, so as to obtain the compensated feature information corresponding to the target image.

[0129] Optionally, the processor 901 is further configured to:

[0130] Obtain a plurality of sample image pairs, where each sample image pair in the plurality of sample image pairs includes a first sample image and a second sample image;

[0131] Use the first feature information of the first sample image and the second feature information corresponding to the second sample image to train the initial model, so as to obtain an image processing model, where the second feature information is the compensated feature information.

[0132] Optionally, the processor 901 is specifically configured to:

[0133] Invoke the feature extraction network included in the initial model to perform feature extraction processing on the first sample image and the second sample image respectively, so as to obtain the first feature information of the first sample image and the first feature information of the second sample image;

[0134] Invoke the feature repair network included in the initial model to perform feature repair processing on the original feature information and the first feature information of the second sample image, so as to obtain the compensated feature information corresponding to the second sample image;

[0135] Determine the second feature information corresponding to the second sample image according to the first feature information of the second sample image and the corresponding compensated feature information;

[0136] Adjust the network parameters of the initial model according to the difference between the first feature information of the first sample image and the second feature information corresponding to the second sample image, so as to obtain an image processing model, where the network parameters include the network parameters of one or both of the feature extraction network and the feature repair network.

[0137] Optionally, the processor 901 is specifically configured to:

[0138] Perform image damage processing on each first sample image in a plurality of first sample images to obtain a second sample image corresponding to each first sample image, where the image damage processing includes one or more of dynamic blur, contrast adjustment, brightness adjustment, adding scratches, image fogging, and adding noise;

[0139] Determine a plurality of sample image pairs according to each first sample image and the corresponding second sample image.

[0140] Optionally, the processor 901 is further configured to:

[0141] Obtain an original image set, where the original image set includes a plurality of original images;

[0142] Obtain a quality evaluation parameter for each original image in the plurality of original images, where the quality evaluation parameter includes one or more of a degree of blurriness, a degree of dimness, and an average difference of image features, and the average difference of image features is the average difference of image features between the original image and the corresponding damaged image;

[0143] Use the original images in the plurality of original images whose quality evaluation parameters meet the image quality screening conditions as first sample images.

[0144] Optionally, the processor 901 is specifically configured to:

[0145] Call the feature extraction network included in the image processing model to obtain first feature information of each original image in the plurality of original images and first feature information of the corresponding damaged image of each original image;

[0146] Determine an average difference of image features between each original image and the corresponding damaged image according to the first feature information of each original image and the first feature information of the corresponding damaged image;

[0147] Use the average difference of the corresponding image features as the quality evaluation parameter of each original image.

[0148] Optionally, the processor 901 is further configured to:

[0149] Use a plurality of first sample images and reference class labels of each first sample image to pre-train the feature extraction network included in the initial model.

[0150] In a specific implementation, the processor 901, the storage device 902, and the network interface 903 described in the embodiments of the present application may execute the implementation manners described in the relevant embodiments of the image processing method provided in the embodiments of the present application Figure 3 or Figure 4 the implementation manners described in the relevant embodiments of the image processing device provided in the embodiments of the present application, which will not be elaborated herein. Figure 8

[0151] ​In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. Essentially, the technical solution of this application, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc., specifically, the processor in the computer device) to execute all or part of the steps of the above methods in the respective embodiments of this application. Among them, the aforementioned storage medium may include: various media that can store program codes such as USB flash drives, mobile hard disks, magnetic disks, optical discs, read-only memory (English: Read-Only Memory, abbreviation: ROM), or random access memory (English: Random Access Memory, abbreviation: RAM).

[0152] As described above, the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the respective embodiments of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain a target image to be processed; Invoke a feature extraction network included in an image processing model to perform feature extraction processing on the target image, and obtain first feature information of the target image, where the first feature information includes high-dimensional feature information; Invoke a feature repair network included in the image processing model to perform feature repair processing on the original feature information and the first feature information of the target image, and obtain compensation feature information corresponding to the target image; the image processing model is trained based on reference feature information of a first sample image and feature information after compensation of a second sample image, and the second sample image is generated based on the first sample image; Determine second feature information corresponding to the target image according to the first feature information and the compensation feature information of the target image, where the second feature information is used to perform classification processing on the target image.

2. The method according to claim 1, characterized in that, Before invoking the feature extraction network included in the image processing model to perform feature extraction processing on the target image and obtain the first feature information of the target image, the method further includes: Obtain a plurality of sample image pairs, where each sample image pair in the plurality of sample image pairs includes a first sample image and a second sample image; Use the first feature information of the first sample image and the second feature information corresponding to the second sample image to train an initial model, and obtain an image processing model, where the second feature information is feature information after compensation.

3. The method according to claim 2, characterized in that The using the first feature information of the first sample image and the second feature information corresponding to the second sample image to train the initial model and obtain the image processing model includes: Invoke the feature extraction network included in the initial model to perform feature extraction processing on the first sample image and the second sample image respectively, and obtain the first feature information of the first sample image and the first feature information of the second sample image; Invoke the feature repair network included in the initial model to perform feature repair processing on the original feature information and the first feature information of the second sample image, and obtain compensation feature information corresponding to the second sample image; Determine the second feature information corresponding to the second sample image according to the first feature information and the corresponding compensation feature information of the second sample image; Adjust network parameters of the initial model according to the difference between the first feature information of the first sample image and the second feature information corresponding to the second sample image, and obtain an image processing model, where the network parameters include network parameters of one or both of the feature extraction network and the feature repair network.

4. The method according to claim 2, characterized in that The obtaining the plurality of sample image pairs includes: Perform image damage processing on each first sample image in the plurality of first sample images to obtain a second sample image corresponding to each first sample image, where the image damage processing includes one or more of dynamic blur, contrast adjustment, brightness adjustment, adding scratches, image fogging, and adding noise; Determine a plurality of sample image pairs according to each first sample image and the corresponding second sample image.

5. The method according to claim 4, wherein The method further includes: Obtain an original image set, where the original image set includes a plurality of original images; Obtain a quality evaluation parameter for each original image in the plurality of original images, where the quality evaluation parameter includes one or more of a blur degree, a dimness degree, and an average difference of image features, and the average difference of image features is the average difference of image features between the original image and the corresponding damaged image; Use the original images in the plurality of original images whose quality evaluation parameters meet the image quality screening conditions as first sample images.

6. The method according to claim 5, wherein The obtaining of the quality evaluation parameter for each original image in the plurality of original images includes: Invoke the feature extraction network included in the image processing model to obtain first feature information of each original image in the plurality of original images and first feature information of the corresponding damaged image of each original image; Determine the average difference of image features between each original image and the corresponding damaged image according to the first feature information of each original image and the first feature information of the corresponding damaged image; Use the corresponding average difference of image features as the quality evaluation parameter of each original image.

7. The method according to claim 3, wherein Before the invoking the feature extraction network included in the initial model to perform feature extraction processing on the first sample image and the second sample image respectively to obtain first feature information of the first sample image and first feature information of the second sample image, the method further includes: Pre-train the feature extraction network included in the initial model by using a plurality of first sample images and the reference class labels of each first sample image.

8. An image processing apparatus, characterized in that, The apparatus includes: An obtaining module, configured to obtain a target image to be processed; A processing module, configured to invoke the feature extraction network included in the image processing model to perform feature extraction processing on the target image to obtain first feature information of the target image, where the first feature information includes high-dimensional feature information; invoke the feature repair network included in the image processing model to perform feature repair processing on the original feature information and the first feature information of the target image to obtain compensated feature information corresponding to the target image; the image processing model is trained based on the reference feature information of the first sample image and the compensated feature information of the second sample image, and the second sample image is generated based on the first sample image; A determining module, configured to determine second feature information corresponding to the target image according to the first feature information of the target image and the compensated feature information, where the second feature information is used to perform classification processing on the target image.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, where the computer program includes program instructions, and the program instructions are executed by a processor to execute the image processing method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

Images