Image processing method, device, electronic device and computer readable storage medium

Abstract

The embodiment of the present application provides an image processing method, device, electronic device and computer-readable storage medium, belonging to the field of image processing, the method comprising: obtaining at least one image feature of an image; enhancing the image feature; obtaining a prediction result according to the enhanced image feature. The image processing method of the present application can effectively improve the efficiency of image processing and reduce the image processing time.

Description

Technical Field

[0001] The present application relates to the field of image processing technology. Specifically, the present application relates to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] Image enhancement is a commonly used technique in machine learning and deep learning, which is a technique that performs geometric transformations (such as translation, rotation, etc.) or color gamut transformations (such as HSV channel enhancement, etc.) on existing images to obtain more image data.

[0003] In the process of image processing, the image is usually enhanced multiple times, and then the enhanced data is input into the network for analysis or calculation, which requires optimization of the existing image processing methods. Summary of the invention

[0004] The purpose of this application is to provide an image processing method, device, electronic device and computer-readable storage medium.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Obtain image features of the image;

[0007] The image features are enhanced; and prediction results are obtained according to the enhanced image features.

[0008] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0009] An acquisition module, used for acquiring image features of an image;

[0010] The enhancement module is used to enhance the image features; and obtain the prediction result according to the enhanced image features.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a processor and a memory:

[0012] a memory for storing machine-readable instructions;

[0013] The processor is used to execute the method shown in the first aspect when executing the above-mentioned machine-readable instructions.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the method shown in the first aspect above is implemented.

[0015] The beneficial effects of the technical solution provided by this application are:

[0016] The above-mentioned image processing method obtains the image features of the image and enhances the image features to obtain the prediction results. In the process of processing the image to obtain the prediction results, the step of obtaining the image features from the image only needs to be performed once, and there is no need to enhance the image features every time and then re-acquire the image features. This can effectively save image processing time and improve image processing efficiency.

[0017] Furthermore, feature enhancement is performed at least twice on the image features output by the network Net, so that the step of calculating the features (i.e., the process of calling the network Net) only needs to be called once; compared with the prior art method of performing enhancement on the image, N-1 times of calculation through the network are saved, thereby greatly saving the calculation time.

[0018] Furthermore, the calculation of the encoder part only needs to be performed once. After obtaining the latent features, each time the features are enhanced, it is only necessary to splice the transform code onto the latent features and perform the decoder calculation to obtain the result of an enhanced feature. Since the network calculation amount of the decoder is much smaller than the calculation amount of the original network Net, the feature enhancement method of the present application is much faster than the original data enhancement method.

[0019] Furthermore, by obtaining a gradient vector through the guidance provided by the network gradient of the first prediction value, a more effective gradient transform coding is obtained, which can achieve a higher accuracy prediction by reducing the number of feature enhancements. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0021] Figure 1 A schematic diagram showing a solution of image processing in the prior art is shown;

[0022] Figure 2 A schematic diagram showing the image processing scheme in the present application;

[0023] Figure 3 A schematic diagram of an image in an example is shown;

[0024] Figure 4 Shows Figure 3 A schematic diagram of a region image of an image in FIG.

[0025] Figure 5 A schematic diagram showing the image processing scheme in the present application;

[0026] Figure 6 A schematic diagram of a process flow of an image processing method provided in an embodiment of the present application is shown;

[0027] Figure 7 A schematic diagram of a process flow of an image processing method provided in an embodiment of the present application is shown;

[0028] Figure 8 A schematic diagram of a process flow of an image processing method provided in an embodiment of the present application is shown;

[0029] Fig. 9 A schematic diagram showing an enhanced solution provided by an embodiment of the present application is shown;

[0030] Fig.10 A schematic diagram of a process flow of an image processing method provided in an embodiment of the present application is shown;

[0031] Fig.11 A schematic diagram of a process flow of an image processing method provided in an embodiment of the present application is shown;

[0032] Fig.12 A schematic diagram of a flow chart of an image processing method provided by an example of the present application is shown;

[0033] Fig.13 A schematic diagram of a flow chart of an image processing method provided by an example of the present application is shown;

[0034] Fig.14 A schematic diagram showing a scheme for training an initial encoder and an initial decoder in an example of the present application is shown;

[0035] Fig.15 A schematic diagram showing the structure of an image processing device provided in an embodiment of the present application is shown;

[0036] Fig.16 A schematic diagram showing the structure of an image processing device provided in an embodiment of the present application is shown;

[0037] Fig.17 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0039] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0040] In order to better understand the solutions provided by the embodiments of the present application, the relevant technologies involved in the present application are briefly described below.

[0041] Data augmentation (image feature enhancement) is a commonly used technique in machine learning and deep learning, which is a technique that performs geometric transformations (such as translation, rotation, etc.) or color gamut transformations (such as HSV channel enhancement, etc.) on existing data to obtain more data.

[0042] A deep learning-based estimation system such as Figure 1 As shown in the figure, the input is usually an image, and the estimation result is obtained through two steps: first, the network Net is used to extract features, and then the predictor is used to predict the output result based on the features. The computational complexity of feature calculation and result prediction is very different, and the typical computational complexity ratio is between 1000:1 and 10000:1. Taking our line of sight estimation as an example, feature calculation requires about 494 trillion calculations, while prediction only requires 0.1 trillion calculations.

[0043] In the test phase, the existing technology performs multiple data enhancement on the input data to obtain multiple data, calculates the network output of multiple data respectively, and averages these outputs to obtain better test results. The specific process is as follows Figure 2 shown.

[0044] The speed performance of existing technologies is so poor that they cannot be used in practical applications. During the testing phase, existing technologies need to calculate the entire model for each enhanced data, that is, after N times of enhancement of a data, N times of model inference is required, and the calculation time is also N times, which is unacceptable for practical applications with speed performance requirements, so the data enhancement algorithm of this testing phase is usually not included in practical applications.

[0045] This application changes the object of the enhancement operation from data to features, and proposes a test framework for enhancing features, which greatly improves the speed compared to the existing data enhancement test framework. This application proposes a feature enhancement network module that encodes transformation information into features, thereby achieving feature enhancement. This application proposes a method for generating efficient features that can reduce the number of enhancements required.

[0046] Below we use the line of sight estimation task to illustrate the invention. However, it should be clear that the invention can be used for other deep learning tasks.

[0047] A typical line of sight estimation system, the framework is as follows Figure 1 As shown in the figure, the input and output are specific data related to the task. Specifically, the input is a face image, such as Figure 3 More precisely, the input is based on the location of facial feature points, from Figure 3 The three sub-images of the face, left eye and right eye regions cut out from Figure 4 The output is the estimated gaze position, which can be the angle of the gaze direction (3D gaze estimation) or the 2D coordinates on the mobile phone screen or monitor screen that the user is looking at (2D gaze estimation).

[0048] As indicated Figure 5 This framework performs N enhancements on the features output by the network Net, so that the step of calculating the features (i.e., the process of calling the network Net) only needs to be called once; compared with the prior art method of enhancing images, this application saves N-1 times of calculations through the network, thereby greatly saving calculation time.

[0049] Taking line of sight estimation as an example, the network consists of 19 convolutional layers and 2 fc layers. One-time expansion: The original image processing method uses 19 convolutional layers and 2 fc layers, which generates 494M calculations. This application uses 2 fc layers (decoders) and 2 fc layers, which generates 0.14M calculations.

[0050] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0051] Figure 6 A schematic diagram of a process of an image processing method provided by an embodiment of the present application is shown, such as Figure 6 As shown in , the method may include the following steps:

[0052] Step S601, obtaining image features of an image.

[0053] The image may be input into a preset network, and the output of a specific layer of the preset network may be used as an image feature, and one image feature or multiple image features may be obtained.

[0054] Taking a face as an example, input a picture like Figure 3 The picture of a complete face taken by the camera shown in the figure is cropped into three pictures of face, left eye and right eye according to the result of facial key point detection. The three pictures are resized to a preset fixed size using bilinear interpolation. For example, the reference sizes are (80x80), (60x80), (60x80), and then the center is cropped to obtain three pictures of face, left eye and right eye with reference sizes of (64x64), (48x64), (48x64), as shown in FIG. Figure 4 shown.

[0055] Then Figure 4 The three pictures of face, left eye and right eye shown in the figure are fed into the network Net to obtain the output features of a specific layer of the network, namely, the image features of the face, the image features of the left eye and the image features of the right eye.

[0056] Step S602, enhancing the image features; obtaining prediction results according to the enhanced image features.

[0057] Among them, enhancement can be performed once or multiple times; in this application, enhancement can also be described as feature enhancement.

[0058] Specifically, in each enhancement process, each of the at least one image feature is enhanced respectively, and a prediction value corresponding to the enhancement is obtained based on the at least one enhanced image feature; the enhancement is repeated multiple times, and corresponding prediction results are obtained based on the prediction values ​​corresponding to the multiple enhancements.

[0059] Taking the face as an example, the image features of the face, the image features of the left eye, and the image features of the right eye are enhanced once respectively to obtain the enhanced image features of the face, the enhanced image features of the left eye, and the enhanced image features of the right eye, and then the prediction value of the enhancement is obtained based on the enhanced image features of the face, the enhanced image features of the left eye, and the enhanced image features of the right eye; if only one enhancement is required, the prediction value is the prediction result; if multiple enhancements are required, the prediction values ​​corresponding to the multiple enhancements are obtained, and the final prediction result is obtained based on the prediction values ​​corresponding to the multiple enhancements. For example, the prediction result can be obtained by taking the average value.

[0060] In the above-mentioned embodiment, by acquiring image features of an image, the image features are enhanced to obtain a prediction result of the image. In the process of processing the image to obtain the prediction result, the step of acquiring image features from the image only needs to be performed once, and there is no need to enhance the image and then re-acquire the image features each time. This can effectively save image processing time and improve image processing efficiency.

[0061] In a possible implementation of the embodiment of the present application, the step S601 of acquiring the image features of the image may include:

[0062] (1) extracting at least two region images from the image; each region image includes a corresponding target object;

[0063] (2) Obtain the image features corresponding to each region image respectively.

[0064] In a specific implementation process, at least two regional images can be input into a preset network to obtain at least two image features corresponding to the output of a specific layer of the preset network; the specific layer is the upper layer of the fully connected layer of the preset network.

[0065] Specifically, the specific target may be a target related to the prediction result, and the regional image may be an image including the specific target; for example, when it is necessary to perform line of sight estimation based on a picture, the specific targets are face, left eye and right eye; the area where the face is located can be extracted from the image as a regional image including the face, the area where the left eye is located can be extracted from the image as a regional image including the left eye, and the area where the right eye is located can be extracted from the image as a regional image including the right eye.

[0066] The preset network may be a partial structure in a preset network. For example, the preset network may be a partial structure in a preset network that does not include a fully connected layer.

[0067] In a specific implementation process, at least two region images are input into a preset network, and for each region image, an image feature corresponding to the image is obtained.

[0068] A possible implementation method of the embodiment of the present application is as follows: Figure 7 As shown, step S602 of enhancing the image features and obtaining the prediction results according to the enhanced image features may include:

[0069] Step S610, obtaining a first enhanced feature obtained after enhancing the image feature.

[0070] Specifically, a first enhanced feature of each image feature obtained after each enhancement is obtained.

[0071] Wherein, for each image feature, the first enhanced feature is obtained by performing one enhancement based on the image feature.

[0072] Specifically, the image feature may be encoded and then decoded to obtain a first enhanced feature. The specific process of each enhancement will be described in detail below.

[0073] Step S620: Obtain a first prediction value of the image based on the first enhanced feature.

[0074] Specifically, based on the first enhancement feature of each image feature, a first prediction value of the image in the enhancement is obtained. Each image feature is enhanced once to obtain a first enhancement feature corresponding to each image feature, and a first prediction value corresponding to the image after the enhancement is obtained based on multiple first enhancement features.

[0075] Step S630: obtaining a prediction result based on the first prediction value.

[0076] Specifically, at least one image feature of the image is enhanced at least twice, a corresponding first prediction value is obtained each time the enhancement is performed, and a prediction result is obtained based on the at least one first prediction value obtained by the at least two enhancements.

[0077] In a possible implementation manner of the embodiment of the present application, the first enhanced feature is obtained based on each image feature after each enhancement, and the step S610 of obtaining the first enhanced feature obtained after enhancing the image feature may include:

[0078] (1) Obtain the latent features of image features.

[0079] Specifically, when each image feature is enhanced each time, an encoder may be used to obtain latent features of the image feature.

[0080] In a specific implementation process, obtaining the latent feature of the image feature may include: inputting the image feature into an encoder for encoding to obtain the latent feature; the encoder includes a first preset number of fully connected layers.

[0081] (2) Obtaining a transformation parameter corresponding to the image feature, and obtaining a first enhanced feature corresponding to the image feature based on the transformation parameter and the latent feature.

[0082] The transformation parameters can be translation, rotation, scaling and other parameters. Take translation parameters as an example: P i ={x:a,y:b} represents replacing the "center crop" in step 1 with "cropping at a pixel to the right of the center and b pixel to the bottom". When a / b is a negative number, it means offset to the left / up.

[0083] Specifically, acquiring the first enhanced feature corresponding to the image feature based on the preset transformation parameter and the latent feature may include:

[0084] a. Convert the transformation parameters into transformation codes, and concatenate the transformation codes and latent features to obtain a first concatenated feature.

[0085] Specifically, the transform parameters can be converted into one-hot encoding to obtain the transform code. For example, the transform code c is a 16-dimensional vector, in which the first eight values ​​are the offset codes of the x-axis, and the last eight values ​​are the offset codes of the y-axis. The one-shot code can know the encoding position according to the offset size. The value at the encoding position is 1, and the remaining seven values ​​are 0.

[0086] b. Decoding the first concatenated feature to obtain a first enhanced feature; the decoder includes a second preset number of fully connected layers.

[0087] Specifically, the first concatenated feature is input into a decoder, and the decoder includes a fully connected layer of a preset network with a second preset number of layers.

[0088] It can be understood that the first preset number of fully connected layers of the encoder and the second preset number of fully connected layers of the decoder may be the same or different.

[0089] like Figure 8 As shown, the image is processed based on a preset network to obtain image features, that is, image features corresponding to at least one feature area in the image, and the image features are enhanced, that is, the image features are input into an encoder for encoding to obtain corresponding latent features, and then transform codes are obtained based on transform parameters. After the latent features and transform codes are concatenated, they are input into a decoder for decoding to obtain feature ' in the figure, that is, a first enhanced feature corresponding to the image features, and then the first enhanced feature is predicted based on a predictor to obtain a prediction result.

[0090] like Fig. 9 As shown, the enhancement part includes an encoder and a decoder. Each time the enhancement is performed, the image features are input into the encoder to obtain the corresponding latent features, and then the latent features and the transform codes are spliced ​​and input into the decoder to obtain the corresponding first enhancement features, that is, Fig. 9 In the feature', when obtaining the prediction result based on the predictor, each enhancement does not need to use the preset network to process the image, therefore, the image processing efficiency can be greatly improved.

[0091] In the specific implementation process, the transform code and latent feature corresponding to each image feature are spliced ​​to obtain a first spliced ​​feature corresponding to each image feature, and then input into the decoder to obtain a first enhanced feature. For example, the transform code is [a, b] and the latent feature is [c, d, e]. The transform code and the latent feature are spliced ​​to obtain the first spliced ​​feature [a, b, c, d, e].

[0092] In a possible implementation of the embodiment of the present application, obtaining a first prediction value of the image based on the first enhanced feature in step S620 may include:

[0093] (1) concatenating the first enhanced features corresponding to at least two image features to obtain a first feature to be predicted in the primary enhancement;

[0094] (2) Obtaining a first prediction value according to the first feature to be predicted.

[0095] In a specific implementation process, the first feature to be predicted may be input into a predictor to obtain a first prediction value corresponding to the primary enhancement.

[0096] Specifically, there is at least one regional image corresponding to the image, and each regional image can obtain a corresponding image feature, and each image feature can obtain a corresponding first enhancement feature, that is, the image can obtain at least one first enhancement feature, and the at least one first enhancement feature is spliced ​​to obtain the first feature to be predicted in the enhancement, and then the first feature to be predicted is input into the predictor to obtain the first prediction value corresponding to the enhancement.

[0097] Take the face as an example, Fig.10 As shown, the image includes three regional images: face image, left eye image and right eye image, and the image features corresponding to the three regional images are obtained respectively, that is, the face image feature, the left eye image feature and the right eye image feature are obtained. The transform coding and latent features corresponding to each image feature are spliced ​​to obtain the first splicing feature corresponding to each image feature, and then input into the decoder to obtain the first enhanced feature, that is, the enhanced face image feature, the enhanced left eye image feature and the enhanced right eye image feature are obtained, that is, Fig.10 The features in ' are concatenated to obtain the first feature to be predicted corresponding to this enhancement, and then the predictor is used to predict to obtain the first predicted value of this enhancement.

[0098] In the present application, there are different ways to obtain the prediction result based on the first prediction value. The following will illustrate the way to obtain the prediction result in combination with the drawings and embodiments.

[0099] In one implementation process, obtaining the prediction result based on the first prediction value in step S630 may include:

[0100] A prediction result is obtained based on at least two first prediction values ​​corresponding to at least two enhancements.

[0101] Specifically, an average value of at least one first prediction value corresponding to at least two enhancements is obtained, and the average value is used as the prediction result.

[0102] Taking the face as an example, the image includes three regional images: face image, left eye image and right eye image. The image features corresponding to the three regional images are obtained respectively, that is, the face image features, the left eye image features and the right eye image features are obtained, and then the three image features are enhanced once respectively to obtain enhanced face image features, enhanced left eye image features and enhanced right eye image features. The enhanced face image features, enhanced left eye image features and enhanced right eye image features are spliced ​​to obtain the first feature to be predicted corresponding to this enhancement, and then the predictor is used to predict to obtain the first prediction value of this enhancement; then multiple enhancements are performed to obtain multiple first prediction values, and the average value of the multiple first prediction values ​​is used as the final prediction result.

[0103] In another implementation process, the transform coding is empty transform coding, that is, the transform coding after the transform parameters corresponding to each image feature are converted into symmetrical empty transform coding; the step S630 of obtaining the prediction result based on the first prediction value may include:

[0104] (1) Based on the first prediction value and the empty transform coding, a plurality of gradient vectors in preset directions are obtained.

[0105] Specifically, for a first prediction value corresponding to a primary enhancement, a gradient for empty transform coding is obtained for the first prediction value to obtain gradient vectors in multiple preset directions.

[0106] In the specific implementation process, the gradient of the first prediction value with respect to the transform code c is calculated respectively. Taking the first prediction value as pred_x, pred_y as an example, the gradient of pred_x, pred_y with respect to the transform code c is calculated respectively, and two gradients in the ascending and descending directions are taken, and a total of 4 gradient vectors are obtained: g_x1, g_x2, g_y1, g_y2. Specifically, taking the ascending direction of pred_x as an example, the calculation formula is:

[0107]

[0108] Where: Indicates finding the gradient, top_1() means retaining the maximum value in the vector and setting the rest to zero. sgn() means taking the sign of each value in the vector, and the output value is {-1,0,1}.

[0109] Similarly, g_x2, g_y1, and g_y2 are calculated in the same way as formula (1).

[0110] (2) For each gradient vector in a preset direction, a second prediction value corresponding to the preset direction is obtained based on the gradient vector, the empty transform coding and the latent feature.

[0111] Specifically, obtaining a second prediction value corresponding to the preset direction based on the gradient vector, the empty transform code, and the latent feature may include:

[0112] a. Add the gradient vector to the empty transform code to obtain the gradient transform code;

[0113] b. Concatenate the gradient transformation code with the latent feature to obtain the second concatenated feature;

[0114] c. Inputting the second concatenated feature into a decoder to obtain a second enhanced feature;

[0115] d. splicing the second enhanced features corresponding to the at least two image features respectively to obtain the second feature to be predicted in the preset direction in the first enhancement;

[0116] e. Inputting the second feature to be predicted into the predictor to obtain a second prediction value of the preset direction of the image corresponding to the primary enhancement.

[0117] In the specific implementation process, each gradient vector is added to the empty transform code to obtain gradient transform codes of multiple preset directions, and then the gradient transform code of each preset direction is spliced ​​with the latent feature to obtain the second splicing feature of each preset direction.

[0118] (3) Based on the second prediction value corresponding to each preset direction, obtain the enhanced prediction result.

[0119] Take the example of an image including three image features, such as Fig.11 As shown, three image features (i.e., the features shown in the figure) are respectively input into the encoder for encoding to obtain three latent features of the three image features, each latent feature is spliced ​​with the preset empty transform code C, and input into the decoder for decoding to obtain the first enhanced feature of each image feature, and then the three first enhanced features are spliced ​​and input into the predictor to obtain the first prediction value; based on the first prediction value and the empty transform code C, the gradient transform codes in four preset directions are obtained, the latent features of each image feature are spliced ​​with the gradient transform codes in four preset directions, and then respectively input into the decoder for decoding to obtain the second enhanced features in four preset directions of each image feature, the second enhanced features of the three image features in one preset direction are spliced ​​and input into the predictor to obtain the second prediction values ​​of the four preset directions corresponding to this enhancement; the average value of the second prediction values ​​of the four preset directions is taken to obtain the prediction result of this enhancement.

[0120] In the above embodiment, the gradient vector is obtained through the guidance provided by the network gradient of the first prediction value, so as to obtain more effective gradient transform coding, which can achieve a higher precision prediction by reducing the number of enhancements.

[0121] The process of obtaining an encoder and a decoder, that is, the process of training an initial encoder and an initial decoder, will be described below in conjunction with the accompanying drawings and embodiments.

[0122] In a possible implementation manner of the embodiment of the present application, the image processing method may further include:

[0123] The sample images are used to train the initial encoder and the initial decoder to obtain the encoder and the decoder.

[0124] Specifically, the sample image features of the sample image are obtained, and the sample image is geometrically transformed to obtain a geometrically transformed image, and then the geometric transformation features of the geometrically transformed image are obtained; the sample image features are input into an initial encoder to obtain corresponding sample latent features, the sample latent features and preset transformation codes are concatenated and input into an initial decoder to obtain a first sample enhanced feature, and the parameters of the initial encoder and the initial decoder are adjusted by supervising the loss function between the first sample enhanced feature and the geometric transformation feature, and the initial encoder and the initial decoder are trained to obtain a trained encoder and decoder.

[0125] The above-mentioned image processing method obtains the image features of the image and enhances the image features to obtain the prediction results of the image. In the process of processing the image to obtain the prediction results, the step of obtaining the image features from the image only needs to be performed once, and there is no need to enhance the image every time and then re-acquire the image features. This can effectively save image processing time and improve image processing efficiency.

[0126] Furthermore, at least one enhancement is performed on the image features output by the network Net, so that the step of calculating the features (i.e., the process of calling the network Net) only needs to be called once; compared with the prior art method of performing enhancement on the image, N-1 times of calculation through the network are saved, thereby greatly saving the calculation time.

[0127] Furthermore, the calculation of the encoder part only needs to be performed once. After obtaining the latent features, each enhancement only requires splicing the transform code to the latent features and performing the decoder calculation to obtain the result of the enhanced features. Since the network calculation amount of the decoder is much smaller than the calculation amount of the original network Net, the enhancement method of the present application is much faster than the original data enhancement method.

[0128] Furthermore, by obtaining a gradient vector through the guidance provided by the network gradient of the first prediction value, a more effective gradient transform coding is obtained, which can achieve a higher precision prediction by reducing the number of enhancements.

[0129] In order to better understand the above-mentioned image processing method, an example of image processing of the present application is described in detail below:

[0130] In Example 1, the image processing method provided by the present application may include the following steps:

[0131] 1) Input a picture data of a complete face taken by a camera. According to the facial key point detection results, the picture is cropped into three pictures of face, left eye and right eye. The three pictures are resized to the preset fixed size by bilinear interpolation. The reference size is (80x80), (60x80), (60x80). Then the central size is cropped to obtain the face X images with the reference size of (64x64), (48x64), (48x64). f , Left Eye X l , right eye X r Three pictures.

[0132] 2) Enter face X f , Left Eye X l , right eye X r The three images are fed to the network Net to obtain the output features F_fe, F_le, and F_re of the specific layer of the network. The output features of the specific layer used in this application are the features before the fully connected layer. In the following steps, F represents F_fe, F_le, and F_re. The subsequent steps of the three features are the same, and the three branches of face, left eye, and right eye are processed respectively.

[0133] 3) The feature F and the N-th transformation parameter {P i , i=1,…,N} is input into the enhancement module to obtain N enhanced features {F T i , i=1,…,N}. The enhancement module will be described in detail in the second part. The transformation parameters can be translation, rotation, scaling and other parameters. In this application, the translation parameter is taken as an example: P i ={x:a,y:b} represents replacing the "center crop" in step 1 with "cropping the center position a pixels to the right and b pixels below". When a / b is a negative number, it means shifting to the left / up. In this application, reference values ​​are given, for the face a, b∈[-8,8], for the left and right eyes a∈[-6,6], b∈[-8,8].

[0134] 4) N ​​enhanced features {F T i , i=1,…,N} input predictor, get N network prediction outputs, which are N two-dimensional coordinates {(x i ,y i ),i=1,…,N}.

[0135] 5) Fuse the N network prediction outputs to obtain the final result. In this application, the fusion adopts the method of calculating the average value, that is, calculating the average point coordinates of N two-dimensional coordinates as the final result (pred_x, pred_y).

[0136] like Fig.12 As shown, Fig.12 Represents an enhancement process to extract the face image X from image X f , left eye image X l and the right eye image X r , the face image X f , left eye image X l and the right eye image X r The output of the specific layer of the network is used as the facial image feature F_fe, the left eye image feature F_le and the right eye image feature F_re, and then based on the transformation parameters P 1 , P 2 and P 3 The facial image feature F_fe, the left eye image feature F_le and the right eye image feature F_re are enhanced to obtain the first enhanced feature of the facial image, the first enhanced feature of the left eye image and the first enhanced feature of the right eye image, and then the first enhanced feature of the facial image, the first enhanced feature of the left eye image and the first enhanced feature of the right eye image are spliced ​​to obtain the first feature to be predicted, and then the first feature to be predicted is predicted to obtain the first predicted value of this enhancement; after multiple enhancements, multiple first predicted values ​​are obtained, and the average value is taken to obtain the final prediction result.

[0137] In the above example, the framework performs N enhancements on the features output by the network Net, so that the step of calculating the features (i.e., the process of calling the network Net) only needs to be called once; compared with the prior art method of enhancing the image, it saves N-1 times of calculations through the network, thereby greatly saving the calculation time.

[0138] In order to better understand the above-mentioned image processing method, an example of image processing of the present application is described in detail below:

[0139] In Example 2, a network module for implementing enhancement is provided, which implements feature enhancement. This module consists of an encoder and a decoder. The encoder encodes the feature vector into latent features, which are concatenated with the transform code and passed through the decoder to obtain the enhanced feature vector.

[0140] Taking line of sight estimation as an example, the specific steps of the enhancement module in the testing phase are as follows:

[0141] 1) The same as step 1) and step 2) in Example 1, the output feature F of the specific layer of the network is obtained. The output feature of the specific layer used in this application is the feature before the fully connected layer. In the following steps, F represents F_fe, F_le, and F_re. The subsequent steps of the three features are consistent, and the three branches of face, left eye, and right eye are processed respectively.

[0142] 2) Input feature F into encoder E to obtain latent feature F_l. In this application, encoder E consists of two fully connected layers with output dimensions of 128 and 64 respectively.

[0143] 3) The transformation parameter P can be a parameter such as translation, rotation, and scaling. In this application, the translation parameter is taken as an example: P = {x:a, y:b} represents replacing the "central crop" in step 1 of the first part with "cropping at a position a pixels to the right and b pixels to the bottom of the central position". When a / b is a negative number, it represents a left / upward offset. In this application, reference values ​​are given, for the face a, b∈[-8,8], for the left and right eyes a∈[-6,6], b∈[-8,8]. The transformation parameter P is converted into a one-hot code as the transformation code c. Specifically, the transformation code c is a 16-dimensional vector, in which the first eight values ​​are the offset codes of the x-axis, and the last eight values ​​are the offset codes of the y-axis. The one-hot code can know the coding position according to the offset size. The value at the coding position is 1, and the remaining seven values ​​are 0.

[0144] 4) Concatenate the latent feature F_l and the transform code c, input them into the decoder D, and output the feature F'. In this application, the decoder D consists of two fully connected layers with output dimensions of 128 and 64 respectively.

[0145] 5) Perform steps 2) to 4) on the eye, left eye, and right eye branches respectively to obtain three feature outputs. Concatenate the three features to obtain a 192-dimensional feature F_tot', which is input into the predictor to obtain the output result.

[0146] 6) For the same test image, multiple transformation parameters P can be applied to execute steps 3) to 5) to obtain multiple output results, and the average value of these results is used as the final result of this test image.

[0147] like Fig.13 As shown, Fig.13 Represents an enhancement process to extract the face image X from image X f , left eye image X l and the right eye image X r , the face image X f , left eye image X l and the right eye image X rThe facial image features F_fe, the left eye image features F_le and the right eye image features F_re are input into the network Net respectively, and the output of the specific layer of the network is used as the facial image features F_fe, the left eye image features F_le and the right eye image features F_re. The facial image features F_fe, the left eye image features F_le and the right eye image features F_re are input into the encoder respectively to obtain the corresponding latent features, that is, the latent features f_fe of the facial image, the latent features f_le of the left eye image, and the latent features f_re of the right eye image shown in the figure. The latent features f_fe of the facial image are compared with the transform code C 1 After splicing, input the decoder to obtain the first enhanced feature F_fe' of the facial image feature; the latent feature f_le of the left eye image is combined with the transform code C 2 After splicing, input the decoder to obtain the first enhanced feature F_le' of the left eye image feature; the latent feature f_re of the right eye image is combined with the transform code C 3 After splicing, input the decoder to obtain the first enhanced feature F_re' of the right eye image feature; splice the three first enhanced features to obtain the feature F_tot', input it into the predictor to obtain the output result of the first enhancement, obtain the output results corresponding to multiple enhancements, and obtain the final prediction result.

[0148] In the above example, the encoder part only needs to be calculated once. After obtaining the latent feature vector, each time we enhance, we only need to concatenate the transform code to the latent feature vector and perform the decoder calculation to obtain an enhanced result. Since the network calculation amount of the decoder is much smaller than the calculation amount of the original network Net, the enhancement method of this application is much faster than the original data enhancement method.

[0149] In order to better understand the above-mentioned image processing method, an example of image processing of the present application is described in detail below:

[0150] In one example three, Fig.14 As shown, the image processing method provided by the present application may include the following steps:

[0151] 1) Carry out steps 1) and 2) of the test phase in Example 2 to obtain the latent feature F_l.

[0152] 2) Take the empty transform code c, concatenate the latent features F_l and c, input the decoder D, and output the feature F'. In this application, the decoder D consists of two fully connected layers with output dimensions of 128 and 64 respectively.

[0153] 3) Concatenate the feature outputs of the face, left eye, and right eye branches to obtain a 192-dimensional feature F_tot', which is input into the predictor to obtain the output prediction result (pred_x, pred_y)

[0154] 4) Calculate the gradient of the prediction results pred_x and pred_y for the transform code c respectively, take the gradient in the ascending and descending directions, and get a total of 4 gradient vectors: g_x1, g_x2, g_y1, g_y2. Specifically, taking the ascending direction of pred_x as an example, the calculation formula is:

[0155]

[0156] in Indicates finding the gradient, top_1() means retaining the maximum value in the vector and setting the rest to zero. sgn() means taking the sign of each value in the vector, and the output value is {-1,0,1}.

[0157] 5) Add the four gradient vectors to the empty transform code c respectively, and concatenate them to the latent feature F_l respectively. After passing through the decoder and predictor, four outputs are obtained, and their average value (pred_avg_x, pred_avg_y) is taken as the final result.

[0158] In the above example, a more efficient transform code c is generated through the guidance provided by the network gradient of the prediction result, so that a higher accuracy prediction can be achieved by reducing the number of enhancements.

[0159] In order to better understand the above-mentioned image processing method, an example of obtaining an encoder and a decoder in this application is described in detail below.

[0160] In one example, the image X is geometrically transformed and compared with the original Figure X Enter the network together to obtain the original feature F and the feature F of the transformed image T After encoding, concatenating, transforming, and decoding the original feature F, the generated feature F' is obtained. T The network is trained using the L2 loss function between F'. Taking line of sight estimation as an example, the specific steps of the enhancement module in the training phase are as follows:

[0161] 1) Select the transformation parameter P. In this application, taking the translation parameter as an example, P = {x:a, y:b} represents replacing the "central crop" in step 1 of the first part with "cropping at a pixel to the right of the central position and b pixel to the bottom". When a / b is a negative number, it means shifting to the left / up. In this application, reference values ​​are given, for the face a, b∈[-8,8], for the left and right eyes a∈[-6,6], b∈[-8,8], and a value is randomly selected.

[0162] 2) According to the transformation parameter P, image X (representing face X) f , Left Eye X l , right eye X r Three pictures) are geometrically transformed to obtain XT , and X and X T are input into the network Net to obtain features F and F T .

[0163] 3) Input feature F into encoder E to obtain hidden feature F_l. In this application, encoder E consists of two fully connected layers with output dimensions of 128 and 64 respectively.

[0164] 4) Convert the transformation parameter P into a one-shot code as the transformation code c. Specifically, the transformation code c is a 16-dimensional vector, in which the first eight values ​​are the offset codes of the x-axis, and the last eight values ​​are the offset codes of the y-axis. The one-shot code can know the coding position according to the offset size. The value at the coding position is 1, and the remaining seven values ​​are 0.

[0165] 5) Concatenate the latent feature F_l and the transform code c, input them into the decoder D, and output the feature F'. In this application, the decoder D consists of two fully connected layers with output dimensions of 128 and 64 respectively.

[0166] 6) Through supervision F T The L2 loss function between and F' is used to train the encoder E and the decoder D. In this application, the parameters of the network Net and the predictor are all part of the pre-trained line of sight estimation network and do not participate in subsequent training.

[0167] The experimental results of the image processing method of the present application on the open source database GAZE_CAPTURE_DB are shown in the following table.

[0168]

[0169] *The time-consuming items in the experiment here are all CPU tests, which calculate the time required for one image and take the average of 30 tests.

[0170] The experimental results of the image processing method of this application on the GazeCN dataset are shown in the following table:

[0171]

[0172]

[0173] The error was tested on the GazeCN dataset, and the STD was tested on the GazeStare dataset, which reflects the robustness. The time cost was tested on the CPU, and the average of the results of 30 random batches was used as the result (batch size 128). The lower the error, STD, and time cost, the better.

[0174] The effect of this application can be seen from the table above:

[0175] 1) Almost no additional time cost: only 5%-12% more than without expansion;

[0176] 2) The time cost of existing work is 569-1506% more than if no working hours were increased.

[0177] 3) Improved accuracy: comparable to existing work.

[0178] 4) Enhanced robustness: STD decreased from 90.89 to 81.40.

[0179] 5) More efficient: Our 10x enhancement effect is almost 25x the existing technology, so in actual use we can use less enhancement time.

[0180] The experimental results of the image processing method of this application on the GazeCapture dataset are shown in the following table:

[0181]

[0182] The error was tested on the GazeCapture dataset, and the time cost was tested on the CPU. The average of 30 random batches was used as the result (batch size 128). The lower the error, STD, and time cost, the better.

[0183] The effect of this application can be seen from the table above:

[0184] 1) Very little additional time cost: only 0.37%-2.83% more than no additional time cost;

[0185] 2) The time cost of existing work is 371%-2425% more than without adding technology.

[0186] 3) Improved accuracy: reduced no-augmentation error by 5.33% and outperformed existing work,

[0187] 4) Enhanced robustness: STD is reduced from 0.9507 to 0.8332.

[0188] 5) Higher efficiency: Our 5x brightness enhancement effect reaches almost 10x, 25x brightness enhancement effect, so less enhancement time can be used in actual use.

[0189] The experimental results of the image processing method of this application on the GazeCapture dataset are shown in the following table:

[0190]

[0191] The error is tested on the GazeCapture dataset, the lower the error, the better. 4x augmentation means 4 gradient directions: x+- and y+-.

[0192] The effect of this application can be seen from the table above:

[0193] 1) Improved accuracy: Reduced 5.6% without increasing the error, and even reduced the error compared to Example 1 of this application.

[0194] 2) More efficient: 4x enhancement (Example 3) outperforms 25x enhancement (Example 1), so we need less time to augment.

[0195] The above embodiment introduces the image processing method from the perspective of the method flow, and the following embodiment introduces it from the perspective of the virtual module, as shown below:

[0196] The present application embodiment provides an image processing device 1500, such as Fig.15 As shown, the device 1500 may include an acquisition module 1501 and an enhancement module 1502, wherein:

[0197] An acquisition module 1501 is used to acquire image features of an image;

[0198] The enhancement module 1502 is used to enhance the image features and obtain the prediction results according to the enhanced image features.

[0199] The above-mentioned image processing device obtains the image features of the image and enhances the image features to obtain the prediction results of the image. In the process of processing the image to obtain the prediction results, the step of obtaining the image features from the image only needs to be performed once, and there is no need to enhance the features of the image every time and then re-acquire the image features, which can effectively save the time of image processing and improve the image processing efficiency.

[0200] In a possible implementation of the embodiment of the present application, when acquiring the image features of an image, the acquisition module 1501 is specifically used to:

[0201] Extracting two region images from the image, each region image including a corresponding specific target;

[0202] The image features corresponding to each region image are obtained respectively.

[0203] In a possible implementation of the embodiment of the present application, the enhancement module 1502 enhances the image features; when obtaining the prediction result according to the enhanced image features, it is specifically used to:

[0204] Obtaining a first enhanced feature obtained after enhancing the image feature;

[0205] Based on the first enhanced feature, obtaining a first prediction value of the image;

[0206] Based on the first prediction value, a prediction result is obtained.

[0207] In a possible implementation of the embodiment of the present application, the enhancement module 1502 obtains the first enhanced feature after enhancing the image feature, and specifically uses:

[0208] Obtaining latent features of image features;

[0209] A transformation parameter corresponding to the image feature is obtained, and a first enhanced feature corresponding to the image feature is obtained based on the transformation parameter and the latent feature.

[0210] In a possible implementation of the embodiment of the present application, when the enhancement module 1502 acquires the first enhancement feature corresponding to the image feature based on the preset transformation parameter and the latent feature, it is specifically used to:

[0211] Convert the transformation parameter into a transformation code, and concatenate the transformation code and the latent feature to obtain a first concatenated feature;

[0212] The first splicing feature is decoded to obtain a first enhanced feature.

[0213] In a possible implementation manner of the embodiment of the present application, the enhancement module 1502 obtains a first prediction value of the image based on the first enhancement feature, specifically for:

[0214] Concatenate the first enhanced features corresponding to the at least two image features to obtain a first feature to be predicted;

[0215] A first prediction value is obtained according to a first feature to be predicted.

[0216] In a possible implementation manner of the embodiment of the present application, the transform code after the transform parameter corresponding to each image feature is converted into a symmetric empty transform code; when the enhancement module 1502 obtains the prediction result based on the first prediction value, it is specifically used to:

[0217] Based on the first prediction value and the empty transform code, obtaining gradient vectors in multiple preset directions;

[0218] For each gradient vector in a preset direction, obtaining a second prediction value corresponding to the preset direction based on the gradient vector, the empty transform coding and the latent feature;

[0219] A prediction result is obtained based on the second prediction value corresponding to each preset direction.

[0220] A possible implementation manner of the embodiment of the present application is to obtain a second prediction value corresponding to the preset direction based on the gradient vector, the empty transform code and the latent feature, specifically for:

[0221] Adding the gradient vector to the empty transform code to obtain the gradient transform code;

[0222] Concatenate the gradient transformation code with the latent feature to obtain a second concatenated feature;

[0223] Decoding the second splicing feature to obtain a second enhanced feature;

[0224] splicing the second enhanced features respectively corresponding to the at least two image features to obtain the second feature to be predicted in the preset direction;

[0225] According to the second feature to be predicted, a second prediction value of the one preset direction is obtained.

[0226] The image processing device for pictures of the embodiments of the present disclosure can execute an image processing method for pictures provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the image processing device for pictures in each embodiment of the present disclosure correspond to the steps in the image processing method for pictures in each embodiment of the present disclosure. For the detailed functional description of each module in the image processing device for pictures, please refer to the description in the corresponding image processing method for pictures shown in the previous text, which will not be repeated here.

[0227] The above introduces the image processing device provided in the embodiment of the present application from the perspective of functional modularization. Next, the image processing device provided in the embodiment of the present application will be introduced from the perspective of hardware instantiation.

[0228] The present application embodiment provides an image processing device 1600, such as Fig.16 As shown, the apparatus 1600 may include a processor 1601, an encoder 1602, a decoder 1603, and a predictor 1604, wherein:

[0229] Processor 1601, used to obtain image features of an image;

[0230] Encoder 1602, used to encode image features;

[0231] A decoder 1603, used for decoding the encoded image features to enhance the image features;

[0232] The predictor 1604 is used to obtain a prediction result according to the enhanced image features.

[0233] The above-mentioned image processing device obtains the image features of the image, encodes and decodes the image features to enhance the image features, thereby obtaining the prediction results. In the process of processing the image to obtain the prediction results, the step of obtaining the image features from the image only needs to be performed once, and there is no need to enhance the features of the image each time and then re-acquire the image features, which can effectively save the time of image processing and improve the image processing efficiency.

[0234] In a possible implementation manner of the embodiment of the present application, when acquiring image features of an image, the processor 1601 is specifically configured to:

[0235] Extracting at least two regional images from the image; each regional image includes a corresponding target object;

[0236] The image features corresponding to each region image are obtained respectively.

[0237] In a possible implementation of the embodiment of the present application, when the encoder 1602 encodes the image features, it is specifically used to:

[0238] Obtain latent features of the image features.

[0239] In a possible implementation manner of the embodiment of the present application, when the decoder 1603 decodes the encoded image features to enhance the image features, it is specifically used to:

[0240] A transformation parameter corresponding to the image feature is obtained, and a first enhanced feature corresponding to the image feature is obtained based on the transformation parameter and the latent feature.

[0241] In a possible implementation manner of the embodiment of the present application, when the decoder 1603 acquires the first enhanced feature corresponding to the image feature based on the transformation parameter and the latent feature, it is specifically configured to:

[0242] Convert the transformation parameter into a transformation code, and concatenate the transformation code and the latent feature to obtain a first concatenated feature;

[0243] The first splicing feature is decoded to obtain a first enhanced feature.

[0244] In a possible implementation manner of the embodiment of the present application, when the predictor 1604 obtains the prediction result according to the enhanced image features, it is specifically used to:

[0245] Based on the first enhanced feature, obtaining a first prediction value of the image;

[0246] Based on the first prediction value, a prediction result is obtained.

[0247] In a possible implementation manner of the embodiment of the present application, when the predictor 1604 obtains the first prediction value of the image based on the first enhanced feature, it is specifically configured to:

[0248] Concatenate the first enhanced features corresponding to the at least two image features to obtain a first feature to be predicted in the primary enhancement;

[0249] A first prediction value is obtained according to a first feature to be predicted.

[0250] In a possible implementation manner of the embodiment of the present application, the transform coding is empty transform coding;

[0251] When the predictor 1604 obtains the prediction result based on the first prediction value, it is specifically used to:

[0252] Based on the first prediction value and the empty transform code, obtaining gradient vectors in multiple preset directions;

[0253] For each gradient vector in a preset direction, obtaining a second prediction value corresponding to the preset direction based on the gradient vector, the empty transform coding and the latent feature;

[0254] A prediction result is obtained based on the second prediction value corresponding to each preset direction.

[0255] In a possible implementation manner of the embodiment of the present application, when the predictor 1604 obtains the second prediction value corresponding to the preset direction based on the gradient vector, the empty transform code and the latent feature, it is specifically used to:

[0256] Adding the gradient vector to the empty transform code to obtain the gradient transform code;

[0257] Concatenate the gradient transformation code with the latent feature to obtain a second concatenated feature;

[0258] Decoding the second concatenated feature to obtain a second enhanced feature;

[0259] splicing the second enhanced features respectively corresponding to the at least two image features to obtain the second feature to be predicted in the preset direction;

[0260] According to the second feature to be predicted, a second prediction value of the one preset direction is obtained.

[0261] The above introduces the image processing device provided in the embodiment of the present application from the perspective of hardware instantiation. Next, the electronic device provided in the embodiment of the present application will be introduced from the perspective of hardware instantiation, and the computing system of the electronic device will be introduced at the same time.

[0262] Based on the same principle as the method shown in the embodiment of the present disclosure, an electronic device is also provided in the embodiment of the present disclosure, which may include but is not limited to: a processor and a memory; the memory is used to store computer operation instructions; the processor is used to execute the image processing method shown in the embodiment by calling the computer operation instructions. Compared with the prior art, the image processing method in the present application can effectively save the time of image processing and improve the efficiency of image processing.

[0263] In an alternative embodiment, an electronic device is provided, such as Fig.17 As shown, Fig.17The electronic device 1700 shown includes: a processor 1701 and a memory 1703. The processor 1701 and the memory 1703 are connected, such as through a bus 1702. Optionally, the electronic device 1700 may also include a transceiver 1704. It should be noted that in actual applications, the transceiver 1704 is not limited to one, and the structure of the electronic device 1700 does not constitute a limitation on the embodiments of the present application.

[0264] Processor 1701 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 1701 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0265] The bus 1702 may include a path to transmit information between the above components. The bus 1702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 1702 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.17 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0266] The memory 1703 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0267] The memory 1703 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 1701. The processor 1701 is used to execute the application code stored in the memory 1703 to implement the contents shown in the above method embodiment.

[0268] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.17 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0269] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run on a computer, the computer can execute the corresponding content of the aforementioned method embodiment. Compared with the prior art, the image processing method in the present application can effectively save image processing time and improve image processing efficiency.

[0270] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0271] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0272] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0273] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0274] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0275] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0276] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, an enhancement module may also be described as a "module for enhancing image features".

[0277] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. An image processing method, characterized in that: include: Obtain image features of the image; Enhance the image features through a neural network to generate enhanced features; generating a prediction result according to the enhanced features; The neural network includes: an encoder and a decoder, and the image features are enhanced by the neural network to generate enhanced features, including: Encoding the image features by the encoder to obtain latent features; Determine a splicing feature by combining the latent feature and the first transform code; The splicing feature is decoded into a first enhanced feature by the decoder.

2. The image processing method according to claim 1, characterized in that: Combining the latent feature with the first transform coding comprises: The latent feature and the first transform code are concatenated.

3. The image processing method according to claim 1, characterized in that: The generating of the enhanced features further comprises: A third enhanced feature is generated based on the image feature and the second transform coding through a neural network.

4. The image processing method according to claim 3, characterized in that: The first transform coding and the second transform coding correspond to different transform parameters.

5. The image processing method according to claim 3, characterized in that: The generating a prediction result according to the enhanced feature comprises: The prediction result is generated by fusing the first prediction value of the first enhanced feature and the first prediction value of the third enhanced feature.

6. The image processing method according to claim 1, characterized in that: The generating a prediction result according to the enhanced feature comprises: generating a plurality of first prediction values ​​based on the plurality of enhanced features; The prediction result is generated by fusing multiple first prediction values.

7. The image processing method according to claim 1, characterized in that: The step of enhancing the image features includes: Image features are enhanced by transform parameters corresponding to different transforms.

8. The image processing method according to claim 7, characterized in that: The transformation includes at least one of scaling, rotation and translation.

9. The image processing method according to claim 1, characterized in that: The acquiring of image features of an image comprises: Extracting at least two regional images from the image; each regional image includes a corresponding target object; The image features corresponding to each region image are obtained respectively.

10. The image processing method according to claim 1, characterized in that: The enhanced features are generated by enhancing the image features; Generate a prediction result according to the enhanced features, including: Acquire a first enhanced feature obtained after enhancing the image feature; Based on the first enhanced feature, obtaining a first prediction value of the image; Based on the first prediction value, the prediction result is obtained.

11. The image processing method according to claim 10, characterized in that: The obtaining of a first enhanced feature obtained by enhancing the image feature includes: Obtaining latent features of the image features; A transformation parameter corresponding to the image feature is obtained, and a first enhanced feature corresponding to the image feature is obtained based on the transformation parameter and the latent feature.

12. The image processing method according to claim 11, characterized in that: The acquiring a first enhanced feature corresponding to the image feature based on the transformation parameter and the latent feature comprises: Converting the transform parameter into a first transform code, and concatenating the first transform code and the latent feature to obtain a first concatenated feature; The first splicing feature is decoded to obtain the first enhanced feature.

13. The image processing method according to any one of claims 10 to 12, characterized in that: The obtaining a first prediction value of the image based on the first enhanced feature includes: Concatenate the first enhanced features corresponding to the at least two image features to obtain a first feature to be predicted; The first prediction value is obtained according to the first feature to be predicted.

14. The image processing method according to claim 12, characterized in that: The first transform code is a null transform code; The acquiring the prediction result based on the first prediction value includes: Based on the first prediction value and the empty transform code, obtaining gradient vectors in multiple preset directions; For each gradient vector in a preset direction, based on the gradient vector, the empty transform code and the latent feature, obtaining a second prediction value corresponding to the preset direction; The prediction result is obtained based on the second prediction value corresponding to each preset direction.

15. The image processing method according to claim 14, characterized in that: The obtaining, based on the gradient vector, the empty transform code and the latent feature, a second prediction value corresponding to the preset direction includes: Adding the gradient vector to the empty transform code to obtain a gradient transform code; Concatenate the gradient transform code with the latent feature to obtain a second concatenated feature; Decoding the second splicing feature to obtain a second enhanced feature; splicing the second enhanced features respectively corresponding to the at least two image features to obtain the second feature to be predicted in the preset direction; A second predicted value of the preset direction is obtained according to the second feature to be predicted.

16. An electronic device, characterized in that: The electronic device comprises a processor and a memory: The memory is used to store machine-readable instructions; The processor is configured to perform the method according to any one of claims 1 to 15 when executing the machine-readable instructions.

17. The electronic device according to claim 16, characterized in that: The electronic device also includes a camera for generating the image.

18. A computer-readable storage medium, characterized in that: The storage medium stores computer instructions, which implement the method according to any one of claims 1 to 15 when executed by a processor.

Images