Machine learning model training method and device, and image processing method and device

Abstract

The disclosure relates to a machine learning model training method and device and an image processing method and device, and relates to the technical field of image processing. The machine learning model training method comprises: expanding at least one pixel of a to-be-processed image sample into a pixel block, the pixel block comprising a plurality of to-be-predicted pixels; processing the pixel block using a first mask to obtain a mask processing result; predicting pixel prediction values of the plurality of to-be-predicted pixels using a machine learning model according to the mask processing result to obtain a processed image sample; and training the machine learning model according to the pixel prediction values of the plurality of to-be-predicted pixels and pixel label values of the plurality of to-be-predicted pixels.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to a machine learning model training method, a machine learning model training apparatus, an image processing method, an image processing apparatus, an electronic device, and a nonvolatile computer readable storage medium. BACKGROUND

[0002] Super-Resolution (SR) is a method of improving the resolution of an original image through hardware or software. The process of obtaining a high-resolution image from a series of low-resolution images is super-resolution reconstruction, which has important application value in many fields.

[0003] In the related art, an up-sampling technique is used to implement super-resolution processing of an image. SUMMARY

[0004] According to some embodiments of the present disclosure, a machine learning model training method is provided, including: expanding at least one pixel of a to-be-processed image sample into a pixel block, the pixel block including a plurality of to-be-predicted pixels; processing the pixel block using a first mask to obtain a mask processing result; predicting pixel prediction values of the plurality of to-be-predicted pixels using a machine learning model according to the mask processing result to obtain a processed image sample; and training the machine learning model according to the pixel prediction values of the plurality of to-be-predicted pixels and pixel label values of the plurality of to-be-predicted pixels.

[0005] In some embodiments, predicting the pixel prediction values of the plurality of to-be-predicted pixels using the machine learning model includes: extracting self-attention features of the pixel block using a self-attention module of the machine learning model; and predicting the pixel prediction values using the machine learning model according to the self-attention features of the pixel block.

[0006] In some embodiments, extracting the self-attention features of the pixel block using the self-attention module of the machine learning model includes: calculating a first self-attention correlation tensor of each to-be-predicted pixel using the self-attention module; processing the first self-attention correlation tensor using a second mask to determine a second self-attention correlation tensor, such that each to-be-predicted pixel cannot learn relevant information of a specified to-be-predicted pixel; and extracting the self-attention features using the self-attention module according to the second self-attention correlation tensor.

[0007] In some embodiments, the calculating, by the self-attention module, the first self-attention correlation tensor of each pixel block comprises: extracting, by the self-attention module, a query vector and a key vector of each to-be-predicted pixel; calculating the first self-attention correlation tensor according to the query vector and the key vector; and processing the first self-attention correlation tensor by the second mask to determine the second self-attention correlation tensor, such that the influence of each to-be-predicted pixel on the specified to-be-predicted pixel is less than a first threshold.

[0008] In some embodiments, the at least one pixel is a seed pixel, and the second self-attention correlation tensor is configured to enable each to-be-predicted pixel to learn only the information of the seed pixel, or each to-be-predicted pixel to learn only its own information and the information of the seed pixel, or the seed pixel to learn the information of each to-be-predicted pixel, each to-be-predicted pixel to learn only the information of the seed pixel, or the seed pixel to learn the information of each to-be-predicted pixel, each to-be-predicted pixel to learn only its own information and the information of the seed pixel.

[0009] In some embodiments, the processing, by the second mask, the first self-attention correlation tensor to determine the second self-attention correlation tensor comprises: determining the second self-attention correlation tensor according to the sum of the second mask and the first self-attention correlation tensor, wherein the elements corresponding to the first correlation relationship in the second mask are 0, and the elements corresponding to the second correlation relationship are less than a second threshold, the first correlation relationship is the correlation relationship between each to-be-predicted pixel and at least one element, and the second correlation relationship is the correlation relationship between each to-be-predicted pixel and the specified to-be-predicted pixel.

[0010] In some embodiments, the first + n x N 2 column of the second mask is a full 0 column, and the remaining elements of the second mask are less than the second threshold, n is 0 or a positive integer, and N is a multiple of the resolution improvement of the processed image sample with respect to the to-be-processed image sample, or the first + n x N 2 column of the second mask is a full 0 column, the diagonal elements of the second mask are 0, and the remaining elements of the second mask are less than the second threshold, or the first + n x N 2 column of the second mask is a full 0 column, and the remaining elements of the second mask are less than the second threshold. 2 row of the second mask is a full 0 row, and the remaining elements of the second mask are less than the second threshold, or the first + n x N 2 column of the second mask is a full 0 column, and the remaining elements of the second mask are less than the second threshold. 2 row of the second mask is a full 0 row, the diagonal elements of the second mask are 0, and the remaining elements of the second mask are less than the second threshold.

[0011] In some embodiments, the self-attention feature of the pixel block is extracted by using the self-attention module of the machine learning model, including: in a case where the training round is less than the round threshold, the self-attention feature is extracted by using the self-attention module according to the second self-attention correlation tensor; in a case where the training round is greater than or equal to the round threshold, the self-attention feature is extracted by using the self-attention module according to the first self-attention correlation tensor.

[0012] In some embodiments, the pixel prediction value of the plurality of to-be-predicted pixels in the pixel block is predicted by using the machine learning model, including: embedding representation of the pixel block is generated by using the machine learning model to generate a first embedding feature tensor; feature information of the pixel block is extracted by using the machine learning model according to the first embedding feature tensor; and the pixel prediction value is predicted by using the machine learning model according to the feature information of the pixel block.

[0013] In some embodiments, the feature information of the pixel block is extracted by using the machine learning model according to the first embedding feature tensor, including: position encoding of the pixels in the image is performed to generate a position encoding tensor; a second embedding feature tensor is generated according to the position encoding tensor and the first embedding feature tensor; and the feature information of the pixels in the image is extracted by using the machine learning model according to the second embedding feature tensor.

[0014] In some embodiments, the position encoding tensor is generated by position encoding the first embedding feature tensor, including: the first embedding feature tensor is divided into a plurality of embedding feature sub-tensors according to different directions; and the plurality of embedding feature sub-tensors are respectively position encoded to generate a plurality of position encoding tensors.

[0015] In some embodiments, the at least one pixel of the to-be-processed image sample is expanded into a pixel block, including: the to-be-processed image sample is divided into a plurality of sub-images; and at least one pixel of the plurality of sub-images is expanded into a pixel block.

[0016] In some embodiments, the to-be-processed image sample is divided into a plurality of sub-images, including: the to-be-processed image is divided into a plurality of image blocks; and each of the plurality of image blocks is divided into a plurality of sub-images for different image channels.

[0017] In some embodiments, the first masks for processing different pixel blocks are all the same, the mask values in the first masks are all the same, or the first masks for processing different pixel blocks are all different, the mask values in each of the first masks are all the same, or the first masks for processing different pixel blocks are all different, the mask values in each of the first masks are all different.

[0018] According to another embodiment of the present disclosure, a processing method of an image is provided, including: expanding at least one pixel of a to-be-processed image sample into a pixel block, the pixel block including a plurality of to-be-predicted pixels; processing the pixel block by using a first mask to obtain a mask processing result; and predicting pixel prediction values of the plurality of to-be-predicted pixels in the pixel block by using a machine learning model according to the mask processing result to obtain a processed image sample, the machine learning model being trained by using the training method in any one of the above embodiments.

[0019] According to still another embodiment of the present disclosure, a training device of a machine learning model is provided, including: an expansion unit configured to expand at least one pixel of a to-be-processed image sample into a pixel block, the pixel block including a plurality of to-be-predicted pixels, and process the pixel block by using a first mask to obtain a mask processing result; a prediction unit configured to predict pixel prediction values of the plurality of to-be-predicted pixels in the pixel block by using a machine learning model according to the mask processing result to obtain a processed image sample; and a training unit configured to train the machine learning model according to the pixel prediction values of the plurality of to-be-predicted pixels and pixel label values of the plurality of to-be-predicted pixels.

[0020] In some embodiments, the prediction unit extracts self-attention features of the pixel block by using a self-attention module of the machine learning model, and predicts the pixel prediction values by using the machine learning model according to the self-attention features of the pixel block.

[0021] In some embodiments, the prediction unit calculates a first self-attention correlation tensor of each to-be-predicted pixel by using the self-attention module, processes the first self-attention correlation tensor by using a second mask to determine a second self-attention correlation tensor, so that each to-be-predicted pixel cannot learn relevant information of a specified to-be-predicted pixel, and extracts the self-attention features by using the self-attention module according to the second self-attention correlation tensor.

[0022] In some embodiments, the prediction unit extracts a query vector and a key vector of each to-be-predicted pixel by using the self-attention module, calculates the first self-attention correlation tensor according to the query vector and the key vector, and processes the first self-attention correlation tensor by using the second mask to determine the second self-attention correlation tensor, so that an influence of each to-be-predicted pixel on a specified to-be-predicted pixel is less than a first threshold.

[0023] In some embodiments, the at least one pixel is a seed pixel, and the second self-attention association tensor is such that each pixel to be predicted can only learn the information of the seed pixel, or each pixel to be predicted can learn its own information and the information of the seed pixel, or the seed pixel can learn the information of each pixel to be predicted, and each pixel to be predicted can only learn the information of the seed pixel, or the seed pixel can learn the information of each pixel to be predicted, and each pixel to be predicted can only learn its own information and the information of the seed pixel.

[0024] In some embodiments, the prediction unit determines the second self-attention association tensor based on the sum of the second mask and the first self-attention association tensor. The elements in the second mask corresponding to the first association relationship are 0, and the elements corresponding to the second association relationship are less than a second threshold. The first association relationship is the association relationship between each pixel to be predicted and at least one element, and the second association relationship is the association relationship between each pixel to be predicted and a specified pixel to be predicted.

[0025] In some embodiments, the (1+n×N)th digit of the second mask 2 The column is all zeros, the remaining elements of the second mask are less than the second threshold, n is 0 or a positive integer, and N is the factor by which the resolution of the processed image sample is improved compared to the image sample to be processed, or the (1+n×N)th element of the second mask. 2 The column is all zeros, the diagonal elements of the second mask are 0, and the remaining elements of the second mask are less than the second threshold, or the (1+n×N)th element of the second mask. 2 The column is all zeros, and the (1+n×N)th column of the second mask 2 A row consisting entirely of zeros, where the remaining elements of the second mask are less than the second threshold, or the (1+n×N)th element of the second mask. 2 The column is all zeros, and the (1+n×N)th column of the second mask 2 The row is all 0s, the diagonal elements of the second mask are 0s, and the remaining elements of the second mask are less than the second threshold.

[0026] In some embodiments, when the number of training epochs is less than a epoch threshold, the prediction unit extracts self-attention features using a self-attention module based on a second self-attention association tensor; when the number of training epochs is greater than or equal to the epoch threshold, the prediction unit extracts self-attention features using a self-attention module based on a first self-attention association tensor.

[0027] In some embodiments, the prediction unit uses a machine learning model to embed the pixel block into a representation and generate a first embedding feature tensor; based on the first embedding feature tensor, it uses a machine learning model to extract feature information of the pixel block; and based on the feature information of the pixel block, it uses a machine learning model to predict the pixel prediction value.

[0028] In some embodiments, the prediction unit positionally encodes the pixels in the image to generate a positionally encoded tensor; generates a second embedding feature tensor according to the positionally encoded tensor and the first embedding feature tensor; and extracts feature information of the pixels in the image by using the machine learning model according to the second embedding feature tensor.

[0029] In some embodiments, the prediction unit divides the first embedding feature tensor into a plurality of embedding feature sub-tensors according to different directions; and positionally encodes the plurality of embedding feature sub-tensors respectively to generate a plurality of positionally encoded tensors.

[0030] In some embodiments, the expansion unit divides the image sample to be processed into a plurality of sub-images; and expands at least one pixel of the plurality of sub-images into a pixel block.

[0031] In some embodiments, the expansion unit divides the image to be processed into a plurality of image blocks; and divides each of the plurality of image blocks into a plurality of sub-images for different image channels.

[0032] In some embodiments, the first masks for processing different pixel blocks are all the same, the mask values in the first masks are all the same, or the first masks for processing different pixel blocks are all different, the mask values in each of the first masks are all the same, or the first masks for processing different pixel blocks are all different, the mask values in each of the first masks are different.

[0033] According to still other embodiments of the present disclosure, there is provided an image processing apparatus, comprising: an expansion unit configured to expand at least one pixel of an image sample to be processed into a pixel block, the pixel block comprising a plurality of to-be-predicted pixels, and process the pixel block by using a first mask to obtain a mask processing result; and a prediction unit configured to predict pixel prediction values of the plurality of to-be-predicted pixels in the pixel block by using a machine learning model according to the mask processing result to obtain a processed image sample, the machine learning model being trained by using the training method of any one of the above embodiments.

[0034] According to still other embodiments of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the training method of the machine learning model or the image processing method according to any one of the above embodiments based on instructions stored in the memory.

[0035] According to still other embodiments of the present disclosure, there is provided a non-volatile computer readable storage medium having stored thereon a computer program, the program being executed by a processor to implement the training method of the machine learning model or the image processing method according to any one of the above embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate exemplary embodiments of the disclosure and together with the description serve to explain the disclosure. In the drawings:

[0037] Figure 1 A flowchart illustrating some embodiments of a method of training a machine learning model of the disclosure;

[0038] Figures 2a-2e A schematic diagram illustrating some embodiments of a method of training a machine learning model of the disclosure;

[0039] Figures 3a-3d A schematic diagram illustrating some embodiments of a second mask of the disclosure;

[0040] Figure 4 A flowchart illustrating some embodiments of a method of training a machine learning model of the disclosure;

[0041] Figure 5 A flowchart illustrating some embodiments of a method of processing an image of the disclosure;

[0042] Figure 6 A block diagram illustrating some embodiments of a training apparatus of a machine learning model of the disclosure;

[0043] Figure 7 A block diagram illustrating some embodiments of a processing apparatus of an image of the disclosure;

[0044] Figure 8 A block diagram illustrating some embodiments of an electronic device of the disclosure;

[0045] Figure 9 A block diagram illustrating some embodiments of an electronic device of the disclosure. DETAILED DESCRIPTION

[0046] Various exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the disclosure unless specifically stated otherwise.

[0047] It should be understood, however, that the sizes of the components shown in the drawings are illustrative and not meant to be limiting.

[0048] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the disclosure or its applications or uses.

[0049] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art(s) can not be discussed in detail in this document, but should be considered as part of the description.

[0050] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary, and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0051] It should be noted that like reference numerals and letters in the various figures indicate similar items, and thus, once any certain item is defined in one figure, it should not have to be discussed further in subsequent figures.

[0052] The inventors of the present disclosure found that the above-mentioned related art has the following problems: when predicting the pixel value of a super-resolution image, information outside the receptive field cannot be learned; and the up-sampling process is not a learnable process, resulting in a decline in the super-resolution processing performance of the image.

[0053] In view of this, the present disclosure proposes a technical solution for training a machine learning model, which can improve the super-resolution processing performance of an image.

[0054] To solve the above technical problems, the present disclosure can complete the super-resolution task through the following embodiments. The image is cut into multiple image blocks, and the to-be-predicted pixels are used to fill the to-be-processed image so that it is equal in size to the image after super-resolution processing; the image features are extracted through a machine learning model, and the purpose of up-sampling is achieved at the same time.

[0055] In this way, by using the to-be-predicted pixels to fill the low-resolution image, the machine learning model combines feature extraction and up-sampling together, solves the technical problem that information outside the receptive field cannot be learned, and makes the image super-resolution effect better. For example, the technical solution of the present disclosure can be implemented through the following embodiments.

[0056] Figure 1 A flowchart showing some embodiments of the training method of the machine learning model of the present disclosure.

[0057] As Figure 1 As shown in step 110, at least one pixel of the to-be-processed image sample is expanded into a pixel block. The pixel block includes multiple to-be-predicted pixels. In some embodiments, the pixels can be expanded according to the resolution of the to-be-processed image and the resolution to be achieved, to generate an expanded image. The size of the expanded image is the same as the size of the image corresponding to the resolution to be achieved.

[0058] In step 120, the pixel block is processed using a first mask to obtain a mask processing result.

[0059] In some embodiments, each pixel in the image to be processed can be used as a seed pixel. An initial value is assigned to each pixel to be tested based on the position of the seed pixel in the pixel block within the first mask and the relationship between the mask values. The seed pixel can be located at any position within the pixel block.

[0060] Figures 2a-2e Schematic diagrams illustrating some embodiments of the training methods for the machine learning models of this disclosure.

[0061] In some embodiments, the seed pixel can learn its own information, and each pixel to be predicted can only learn information from the seed pixel. For example, the seed pixel is in the first row and first column of each pixel block, and the first mask is the same for processing different pixel blocks, with all mask values ​​being the same.

[0062] like Figure 2a As shown, the size of the image to be processed is m×n, and the size of the first mask is N×N, where N is the factor by which the resolution of the processed image sample is improved compared to the image sample to be processed. The seed pixels in the first mask are located in 1 row and 1 column, and all mask values ​​are the same (all are masks).

[0063] In some embodiments, the seed pixel can learn its own information, and each pixel to be predicted can learn its own information and the seed pixel's information. For example, the seed pixel is in the first row and first column of each pixel block, and the first mask is the same for processing different pixel blocks, but the mask values ​​in the first mask are different from each other.

[0064] like Figure 2b As shown, all seed pixels share a single N×N first mask. The seed pixels in the first mask are located in 1 row and 1 column, and the mask values ​​at different positions are all different. i,j ,i=1,2…N, j=1,2…N).

[0065] In some embodiments, the seed pixel can learn information from each pixel to be predicted, and each pixel to be predicted can only learn information from the seed pixel. For example, the seed pixel is in the first row and first column of each pixel block, and the first mask for processing different pixel blocks is different, while the mask values ​​in each first mask are the same.

[0066] like Figure 2c As shown, different seed pixels use different N×N size first masks. In each first mask, the seed pixels are located in 1 row and 1 column, and the mask values ​​are the same at different positions. However, the pixel values ​​of the pixels to be predicted in different first masks are different. For example, the pixel value in the first mask corresponding to seed pixel (1,1) is mask. 1,1 The pixel value in the first mask corresponding to the seed pixel (1, n) is mask. 1,n .

[0067] In some embodiments, the seed pixels can learn the information of each to-be-predicted pixel, and each to-be-predicted pixel can only learn its own information and the information of the seed pixels. For example, the seed pixels are in the first row and the first column of each pixel block, and the first masks for processing different pixel blocks are all different, and the mask values in each of the first masks are all different.

[0068] As shown in Figure 2d , different seed pixels use different first masks of N x N size. The seed pixels are in the first row and the first column in each of the first masks, and the mask values in different positions are all different. The pixel values in the first mask corresponding to the seed pixel (1, 1) are mask11 i,j , i = 1, 2…N, j = 1, 2…N, and the pixel values in the first mask corresponding to the seed pixel (1, n) are mask1n i,j .

[0069] In some embodiments, the to-be-processed image samples are divided into a plurality of sub-images, and at least one pixel of the plurality of sub-images is expanded into a pixel block.

[0070] In some embodiments, the to-be-processed image is divided into a plurality of image blocks, and each of the plurality of image blocks is divided into a plurality of sub-images for different image channels. For example, the division of the sub-images can be implemented by the embodiments in Figure 2a .

[0071] As shown in Figure 2e , the to-be-processed image is segmented and divided into a plurality of image blocks, and for an image block, sub-images are generated on the R (red), G (green), and B (blue) channels, and subsequent processing is performed.

[0072] In some embodiments, the to-be-processed image can also not be cut, but the entire to-be-processed image is divided into sub-images of three channels, and subsequent processing is performed.

[0073] In this way, by segmenting the to-be-processed image into smaller sub-images and then performing subsequent processing, the machine learning model needs to learn less useless information, and the required computing resources are also less, thereby improving the processing efficiency.

[0074] In step 130, the pixel prediction values of the plurality of to-be-predicted pixels are predicted by using the machine learning model, and the processed image samples are obtained.

[0075] In some embodiments, the self-attention features of the pixel block are extracted by using the self-attention module of the machine learning model, and the pixel prediction values are predicted by using the machine learning model according to the self-attention features of the pixel block.

[0076] In some embodiments, the pixel block can be unfolded into a one-dimensional vector before being processed by the embedding representation process to obtain the first embedding feature tensor. For example, the pixel block can also be directly processed by the embedding representation process without being unfolded.

[0077] For example, if unfolded into a one-dimensional vector, each pixel in the augmented image can be processed by the embedding representation process to obtain an embedding feature tensor. The embedding dimension can be D, and an embedding feature tensor of mn x D can be obtained. The embedding feature tensor is input into the self-attention module of the encoder of the transformer model to extract self-attention features of size mn x D.

[0078] In some embodiments, the encoder can include a multi-head self-attention module and a feedforward neural network module. The multi-head self-attention module can extract multiple groups of feature vector groups composed of query (Query) vectors, key (Key) vectors, and value (Value) vectors, and then obtain multi-head self-attention features; the multi-head self-attention features can be fused into final self-attention features.

[0079] For example, for each head, the embedding feature tensor is subjected to three linear transformations to generate query vectors, key vectors, and value vectors; the query vector of each pixel is multiplied by the transpose of the key vector of itself and other pixels to calculate the self-attention correlation tensor of each pixel; the self-attention correlation tensor of each pixel is divided by After the softmax process, d is the number of dimensions of the key vector; the softmax processing result of each pixel is multiplied by the value vector of itself and other pixels, respectively, and the sum of the products is calculated; based on the sum result, the self-attention feature of each pixel is determined.

[0080] In some embodiments, the first self-attention correlation tensor of each to-be-predicted pixel is calculated using the self-attention module; the second self-attention correlation tensor is determined by processing the first self-attention correlation tensor using the second mask, so that each to-be-predicted pixel cannot learn the relevant information of the specified to-be-predicted pixel; the self-attention feature is extracted using the self-attention module based on the second self-attention correlation tensor.

[0081] In the above embodiments, considering that the initial prediction value of the to-be-predicted pixel is randomly initialized and does not contain useful information during the training process, it may affect the learning effect of other to-be-predicted pixels. By setting the second mask, the to-be-predicted pixel can not learn the useless information of other to-be-predicted pixels at this stage, but can learn the useful information of the seed pixel, thereby improving the super-resolution processing performance.

[0082] In some embodiments, the query vector and the key vector of each pixel to be predicted are extracted by using the self-attention module; the first self-attention correlation tensor is calculated according to the query vector and the key vector; the first self-attention correlation tensor is processed by using the second mask to determine the second self-attention correlation tensor, so that the influence of the query vector of each pixel to be predicted on the key vector of the specified pixel to be predicted is less than the first threshold.

[0083] For example, the first self-attention correlation tensor is processed by using the second mask to determine the second self-attention correlation tensor; after the second self-attention correlation tensor is processed by using the softmax, the influence of each pixel to be predicted on the specified pixel to be predicted is 0.

[0084] For example, the first self-attention correlation tensor is processed by using the second mask and then processed by using the softmax to obtain the second self-attention correlation tensor.

[0085] For example, the second mask also makes the element in the second self-attention correlation tensor representing the influence of the query vector of each pixel to be predicted on the key vector of the seed pixel consistent with the corresponding element in the first self-attention correlation tensor, so as to retain the influence of the query vector of each pixel to be predicted on the key vector of the seed pixel.

[0086] Figures 3a-3d A schematic diagram showing some embodiments of the second mask of the present disclosure.

[0087] As shown in Figure 3a , the 1+n×N 2 column of the second mask is a full 0 column, the remaining elements of the second mask are less than the second threshold, n is 0 or a positive integer, and N is the multiple of the resolution increase of the processed image sample with respect to the image sample to be processed. For example, the first threshold can be equal to the second threshold, such as both being negative infinity.

[0088] As shown in Figure 3b , the 1+n×N 2 column of the second mask is a full 0 column, the diagonal elements of the second mask are 0, and the remaining elements of the second mask are less than the second threshold.

[0089] As shown in Figure 3c , the 1+n×N 2 column of the second mask is a full 0 column, the 1+n×N 2 row of the second mask is a full 0 row, and the remaining elements of the second mask are less than the second threshold.

[0090] As shown in Figure 3d , the 1+n×N 2 column of the second mask is a full 0 column, the 1+n×N 2The behavior of the all-0 row is that the diagonal elements of the second mask are 0, and the remaining elements of the second mask are negative numbers less than the second threshold (for example, these remaining elements can be -1 x 10 9 or negative infinity).

[0091] In some embodiments, after the one-dimensional embedding representation processing of the pixel block, the first self-attention correlation tensor can be obtained by using a machine learning model; then the first self-attention correlation tensor is processed by using the second mask in Figures 3a-3d , to obtain the second self-attention correlation tensor.

[0092] For example, after the two-dimensional embedding representation processing of the pixel block, the first self-attention correlation tensor can be obtained by using a machine learning model; then the first self-attention correlation tensor is processed by using the second mask corresponding to the two-dimensional embedding representation processing, to obtain the second self-attention correlation tensor.

[0093] In some embodiments, the second self-attention correlation tensor is determined according to the sum of the second mask and the first self-attention correlation tensor, the elements corresponding to the first correlation relationship in the second mask are 0, and the elements corresponding to the second correlation relationship are less than the second threshold (for example, these elements can be negative infinity), the first correlation relationship is the correlation relationship between each to-be-predicted pixel and the seed element, and the second correlation relationship is the correlation relationship between each to-be-predicted pixel and the specified to-be-predicted pixel.

[0094] For example, the first self-attention correlation tensor is added to the second mask to obtain the second self-attention correlation tensor. The second self-attention correlation tensor is processed by scale (i.e., divided by ) to prevent data from being too large, causing gradient disappearance or gradient explosion. After the softmax processing, the value vector is multiplied. The output value (such as the self-attention feature) of the self-attention processing module is added to the input value of the self-attention processing module, and after the layer normalization processing, the output of the multi-head self-attention module is obtained.

[0095] In some embodiments, the second self-attention correlation tensor is determined according to the subtraction result of the first self-attention correlation tensor and the second mask, and the elements corresponding to the first correlation relationship in the second mask are 0, and the elements corresponding to the second correlation relationship are greater than the second threshold (for example, these elements can be positive infinity).

[0096] In some embodiments, the output of the multi-head self-attention module can be processed by using a feedforward neural network; then the processing result is added to the output of the multi-head self-attention module, and after the layer normalization processing, the output of the entire encoder layer is obtained.

[0097] In some embodiments, when the training round is less than the round threshold, the self-attention feature is extracted according to the second self-attention correlation tensor by using the self-attention module; and when the training round is greater than or equal to the round threshold, the self-attention feature is extracted according to the first self-attention correlation tensor by using the self-attention module.

[0098] For example, in the training process, the training can be divided into two stages. In the first stage (for example, the first K rounds of training), the predicted pixel value is relatively accurate, and the second mask is used for training to improve the accuracy; in the second stage, the predicted pixel is relatively accurate, and the second mask is not used for training, and each pixel improves the accuracy by learning from each other.

[0099] For example, in the entire training process, there can be only the first stage, only the second stage, or both stages.

[0100] Figure 4 A flowchart showing another embodiment of the training method of the machine learning model of the present disclosure.

[0101] As shown in Figure 4 The feature extraction of the to-be-processed image sample can be performed by using a (CNN) convolutional neural network to obtain a to-be-predicted image. The to-be-processed image can also be directly processed as a to-be-predicted image for subsequent processing without using a CNN.

[0102] In some embodiments, the pixels in the image can be subjected to position embedding processing. For example, the pixels in the image are subjected to embedding representation processing by using a machine learning model to generate a first embedding feature tensor; and the feature information of the pixels in the image is extracted by using a machine learning model according to the first embedding feature tensor; for example, the image can also be divided into multiple image blocks, and the pixels in the image blocks are subjected to the above position embedding processing.

[0103] In some embodiments, the pixels in the image are subjected to position encoding to generate a position encoding tensor; a second embedding feature tensor is generated according to the position encoding tensor and the first embedding feature tensor; the feature information of the pixels in the image is extracted by using a machine learning model according to the second embedding feature tensor; and the processed image sample is obtained according to the feature information, which is used for training the machine learning model.

[0104] For example, the position encoding can be one-dimensional position encoding, such as absolute position encoding or relative position encoding.

[0105] In some embodiments, for a first embedding feature tensor of size mn×D, where D is the number of dimensions for the embedding representation processing, the first D / 2 of the first embedding feature tensor can be encoded as the X direction, and the last D / 2 of the first embedding feature tensor can be encoded as the Y direction, resulting in embedding feature sub-tensors Ex and Ey of size mn×D / 2, respectively; Ex and Ey are concatenated together to form a positional encoding Epos of size mn. × D;

[0106] For example, the X and Y directions are encoded as Ex and Ey respectively, both with a size of mn. × D. Add Ex and Ey to form the position code Epos.

[0107] In this way, by using positional encoding, the positional information of pixels can be incorporated into the pixel prediction process, thereby improving the accuracy of pixel prediction.

[0108] In step 140, a machine learning model is trained based on the pixel prediction values ​​and pixel label values ​​of multiple pixels to be predicted.

[0109] In some embodiments, the MSE (mean squared error) loss function or the cross-entropy loss function can be used to train a machine learning model.

[0110] In some embodiments, the resolution increase factor N before and after super-resolution processing can be divided into N1 and N2, i.e., N = N1 × N2. By successively replacing N in any of the above embodiments with N1 and N2, and performing staged super-resolution processing and training, the same resolution can be obtained, and the problem of low processing efficiency caused by excessive tensor sparsity during the calculation process can be avoided. For example, N can also be divided into more than two parts, i.e., N = N1 × N2 × … × Nn, and the processing method is similar to that in the above embodiments.

[0111] In the above embodiments, multiple pixels to be predicted are used to expand the existing pixels in the image to be processed, and a machine learning model is used to predict the pixel values ​​of these pixels. This allows information outside the receptive field to be learned, and the prediction process is learnable, thereby improving the accuracy of pixel prediction results and thus enhancing the super-resolution processing performance of the image.

[0112] Figure 5 Flowcharts illustrating some embodiments of the image processing methods of this disclosure are shown.

[0113] like Figure 5 As shown, in step 510, at least one pixel of the image sample to be processed is expanded into a pixel block, and the pixel block includes multiple pixels to be predicted.

[0114] In step 520, the pixel block is processed using the first mask to obtain the mask processing result.

[0115] In step 530, according to the mask processing result, pixel prediction values of the plurality of to-be-predicted pixels in the pixel block are predicted by using the machine learning model, to obtain a processed image sample. The machine learning model is trained by using the training method in any one of the above embodiments.

[0116] Figure 6 A block diagram of some embodiments of the training apparatus of the machine learning model of the present disclosure is shown.

[0117] As shown in Figure 6 The training apparatus 6 of the machine learning model includes an expansion unit 61 configured to expand at least one pixel of a to-be-processed image sample into a pixel block, the pixel block including a plurality of to-be-predicted pixels, process the pixel block by using a first mask to obtain a mask processing result; a prediction unit 62 configured to predict pixel prediction values of the plurality of to-be-predicted pixels in the pixel block by using the machine learning model according to the mask processing result, to obtain a processed image sample; and a training unit 63 configured to train the machine learning model according to the pixel prediction values of the plurality of to-be-predicted pixels and pixel label values of the plurality of to-be-predicted pixels.

[0118] In some embodiments, the prediction unit 62 extracts self-attention features of the pixel block by using a self-attention module of the machine learning model; and predicts the pixel prediction values by using the machine learning model according to the self-attention features of the pixel block.

[0119] In some embodiments, the prediction unit 62 calculates a first self-attention correlation tensor of each to-be-predicted pixel by using the self-attention module; processes the first self-attention correlation tensor by using a second mask to determine a second self-attention correlation tensor, so that each to-be-predicted pixel cannot learn the related information of a specified to-be-predicted pixel; and extracts the self-attention features by using the self-attention module according to the second self-attention correlation tensor.

[0120] In some embodiments, the prediction unit 62 extracts a query vector and a key vector of each to-be-predicted pixel by using the self-attention module; calculates the first self-attention correlation tensor according to the query vector and the key vector; and processes the first self-attention correlation tensor by using the second mask to determine the second self-attention correlation tensor, including processing the first self-attention correlation tensor by using the second mask to determine the second self-attention correlation tensor, so that the influence of each to-be-predicted pixel on a specified to-be-predicted pixel is less than a first threshold.

[0121] In some embodiments, the at least one pixel is a seed pixel, and the seed pixel is capable of learning its own information. The second self-attention correlation tensor is configured to enable each to-be-predicted pixel to learn only the information of the seed pixel, or each to-be-predicted pixel to learn its own information and the information of the seed pixel, or the seed pixel to learn the information of each to-be-predicted pixel, each to-be-predicted pixel to learn only the information of the seed pixel, or the seed pixel to learn the information of each to-be-predicted pixel, each to-be-predicted pixel to learn its own information and the information of the seed pixel.

[0122] In some embodiments, the prediction unit 62 determines the second self-attention correlation tensor according to a sum of the second mask and the first self-attention correlation tensor, an element corresponding to the first correlation relationship in the second mask is 0, and an element corresponding to the second correlation relationship in the second mask is less than a second threshold value, the first correlation relationship is a correlation relationship between each to-be-predicted pixel and at least one element, and the second correlation relationship is a correlation relationship between each to-be-predicted pixel and a specified to-be-predicted pixel.

[0123] In some embodiments, the first + n x N 2 column of the second mask is a full 0 column, and the remaining elements of the second mask are less than the second threshold value, n is 0 or a positive integer, and N is a multiple of the resolution increase of the processed image sample with respect to the to-be-processed image sample, or the first + n x N 2 column of the second mask is a full 0 column, the diagonal elements of the second mask are 0, and the remaining elements of the second mask are less than the second threshold value, or the first + n x N 2 column of the second mask is a full 0 column, and the remaining elements of the second mask are less than the second threshold value. 2 column of the second mask is a full 0 column, and the remaining elements of the second mask are less than the second threshold value. 2 column of the second mask is a full 0 column, and the remaining elements of the second mask are less than the second threshold value. 2 column of the second mask is a full 0 column, and the remaining elements of the second mask are less than the second threshold value.

[0124] In some embodiments, the prediction unit 62 extracts the self-attention feature by using the self-attention module according to the second self-attention correlation tensor when the number of training rounds is less than a round threshold value, and extracts the self-attention feature by using the self-attention module according to the first self-attention correlation tensor when the number of training rounds is greater than or equal to the round threshold value.

[0125] In some embodiments, the prediction unit 62 uses a machine learning model to perform embedding representation on the pixel block to generate a first embedding feature tensor, extracts feature information of the pixel block by using the machine learning model according to the first embedding feature tensor, and predicts the pixel prediction value by using the machine learning model according to the feature information of the pixel block.

[0126] In some embodiments, the prediction unit 62 positionally encodes the pixels in the image to generate a positionally encoded tensor; generates a second embedding feature tensor according to the positionally encoded tensor and the first embedding feature tensor; and extracts feature information of the pixels in the image by using the machine learning model according to the second embedding feature tensor.

[0127] In some embodiments, the prediction unit 62 divides the first embedding feature tensor into a plurality of embedding feature sub-tensors according to different directions; and respectively positionally encodes the plurality of embedding feature sub-tensors to generate a plurality of positionally encoded tensors.

[0128] In some embodiments, the augmentation unit 61 divides the image sample to be processed into a plurality of sub-images; and augments at least one pixel of the plurality of sub-images into a pixel block.

[0129] In some embodiments, the augmentation unit 61 divides the image to be processed into a plurality of image blocks; and divides each of the plurality of image blocks into a plurality of sub-images for different image channels.

[0130] In some embodiments, the first masks for processing different pixel blocks are all the same, the mask values in the first masks are all the same, or the first masks for processing different pixel blocks are all different, the mask values in each of the first masks are all the same, or the first masks for processing different pixel blocks are all different, the mask values in each of the first masks are different.

[0131] Figure 7 A block diagram illustrating some embodiments of an image processing apparatus of the present disclosure.

[0132] As shown in Figure 7 The image processing apparatus 7 includes an augmentation unit 71 configured to augment at least one pixel of an image sample to be processed into a pixel block, the pixel block including a plurality of to-be-predicted pixels, and process the pixel block by using a first mask to obtain a mask processing result; and a prediction unit 72 configured to predict pixel prediction values of the plurality of to-be-predicted pixels in the pixel block by using a machine learning model according to the mask processing result to obtain a processed image sample, the machine learning model being trained by using the training method of any one of the above embodiments.

[0133] Figure 8 A block diagram illustrating some embodiments of an electronic device of the present disclosure.

[0134] As shown in Figure 8 The apparatus 8 of this embodiment includes a memory 81 and a processor 82 coupled to the memory 81, the processor 82 being configured to execute the training method of the machine learning model or the image processing method in any one of the embodiments of the present disclosure based on instructions stored in the memory 51.

[0135] The memory 81 can include, for example, a system memory, a fixed nonvolatile memory medium, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader, a database, and other programs.

[0136] Figure 9 A block diagram illustrating another embodiment of the electronic device of the present disclosure.

[0137] As Figure 9 shown, the electronic device 9 of this embodiment includes a memory 910 and a processor 920 coupled to the memory 910, the processor 920 being configured to perform the training method of the machine learning model or the processing method of the image in any one of the preceding embodiments based on instructions stored in the memory 910.

[0138] The memory 910 can include, for example, a system memory, a fixed nonvolatile memory medium, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader, and other programs.

[0139] The electronic device 9 can further include an input / output interface 930, a network interface 940, a storage interface 950, and the like. These interfaces 930, 940, 950, and the memory 910 and the processor 920 can be connected through a bus 960, for example. The input / output interface 630 provides a connection interface for display, mouse, keyboard, touch screen, microphone, speaker, and other input / output devices. The network interface 940 provides a connection interface for various networking devices. The storage interface 950 provides a connection interface for external storage devices such as SD card and U disk.

[0140] Those skilled in the art will appreciate that embodiments of the present disclosure can be provided as a method, system, or computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media including, but not limited to, disk storage, CD-ROMs, optical storage, and the like.

[0141] Thus far, the training method of the machine learning model, the training apparatus of the machine learning model, the processing method of the image, the processing apparatus of the image, the electronic device, and the non-transitory computer-readable storage medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0142] The methods and systems of the present disclosure can be implemented in a number of ways. For example, the methods and systems of the present disclosure can be implemented via software, hardware, firmware, or any combination of software, hardware, and firmware. The above described order of steps for the methods is merely for illustration, and the steps of the methods of the present disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present disclosure can also be implemented as a program recorded in a recording medium, which includes machine readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the methods according to the present disclosure.

[0143] While certain specific embodiments of the present disclosure have been described in detail by way of example, it should be understood that the examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. It should be appreciated that modifications to the above embodiments can be made which do not depart from the spirit and scope of the present disclosure. The scope of the present disclosure is defined by the claims appended hereto.

Claims

1. A method for training a machine learning model, comprising: At least one pixel of the image sample to be processed is expanded into a pixel block, the pixel block comprising multiple pixels to be predicted; The pixel block is processed using the first mask to obtain the mask processing result; Based on the mask processing result, the machine learning model is used to predict the pixel prediction values ​​of the multiple pixels to be predicted, and the processed image sample is obtained. The machine learning model is trained based on the pixel prediction values ​​and pixel annotation values ​​of the plurality of pixels to be predicted. Wherein, at least one pixel is a seed pixel, and the process of processing the pixel block using a first mask to obtain the mask processing result includes: Based on the arbitrary position of the seed pixel in the pixel block in the first mask, and the relationship between the mask values ​​in the first mask, each of the plurality of pixels to be predicted is assigned a corresponding initial value, so as to obtain the plurality of pixels to be predicted with the initial value as the mask processing result. The step of predicting the pixel prediction values ​​of the plurality of pixels to be predicted using the machine learning model based on the mask processing result includes: Based on the initial value corresponding to each pixel to be predicted, the machine learning model is used to predict the pixel prediction value of the plurality of pixels to be predicted.

2. The training method according to claim 1, wherein, The step of using the machine learning model to predict the pixel prediction values ​​of the plurality of pixels to be predicted includes: The self-attention module of the machine learning model is used to extract the self-attention features of the pixel block; Based on the self-attention features of the pixel block, the machine learning model is used to predict the pixel prediction value.

3. The training method according to claim 2, wherein, The step of extracting the self-attention features of the pixel block using the self-attention module of the machine learning model includes: Using the self-attention module, the first self-attention association tensor of each pixel to be predicted is calculated; The first self-attention association tensor is processed using a second mask to determine the second self-attention association tensor, so that each pixel to be predicted cannot learn the relevant information of the specified pixel to be predicted. Based on the second self-attention association tensor, the self-attention features are extracted using the self-attention module.

4. The training method according to claim 3, wherein, The step of using the self-attention module to calculate the first self-attention association tensor of each pixel block includes: Using the self-attention module, the query vector and key vector of each pixel to be predicted are extracted; Calculate the first self-attention association tensor based on the query vector and the key vector; The step of processing the first self-attention association tensor using the second mask to determine the second self-attention association tensor includes: The first self-attention association tensor is processed using the second mask to determine the second self-attention association tensor, such that the influence of each pixel to be predicted on the specified pixel to be predicted is less than a first threshold.

5. The training method according to claim 4, wherein, The second self-attention association tensor makes, Each pixel to be predicted can only learn information from the seed pixel, or Each pixel to be predicted can learn its own information and the information of the seed pixel, or The seed pixel can learn information from each pixel to be predicted, and each pixel to be predicted can only learn information from the seed pixel, or The seed pixel can learn information from each pixel to be predicted, while each pixel to be predicted can only learn its own information and the information from the seed pixel.

6. The training method according to claim 4, wherein, The step of processing the first self-attention association tensor using the second mask to determine the second self-attention association tensor includes: The second self-attention association tensor is determined by summing the second mask and the first self-attention association tensor. The element in the second mask corresponding to the first association relationship is 0, and the element corresponding to the second association relationship is less than the second threshold. The first association relationship is the association relationship between each pixel to be predicted and the at least one element, and the second association relationship is the association relationship between each pixel to be predicted and the specified pixel to be predicted.

7. The training method according to claim 6, wherein, The (1+n)th of the second mask The column is all zeros, the remaining elements of the second mask are less than the second threshold, n is 0 or a positive integer, and N is the factor by which the resolution of the processed image sample is improved compared to the resolution of the image sample to be processed. The (1+n)th of the second mask The column is all zeros, the diagonal elements of the second mask are 0, and the remaining elements of the second mask are less than the second threshold, or The (1+n)th of the second mask The column is all zeros, and the (1+n)th column of the second mask The row is all 0s, and the remaining elements of the second mask are less than the second threshold, or The (1+n)th of the second mask The column is all zeros, and the (1+n)th column of the second mask The row is all 0s, the diagonal elements of the second mask are 0s, and the remaining elements of the second mask are less than the second threshold.

8. The training method according to claim 3, wherein, The step of extracting the self-attention features of the pixel block using the self-attention module of the machine learning model includes: When the number of training rounds is less than the round threshold, the self-attention features are extracted using the self-attention module based on the second self-attention association tensor. If the number of training rounds is greater than or equal to the round threshold, the self-attention features are extracted using the self-attention module based on the first self-attention association tensor.

9. The training method according to claim 1, wherein, The step of using the machine learning model to predict the pixel prediction values ​​of multiple pixels to be predicted in the pixel block includes: Using the machine learning model, the pixel block is embedded to generate a first embedding feature tensor; Based on the first embedded feature tensor, the machine learning model is used to extract the feature information of the pixel block; Based on the feature information of the pixel block, the machine learning model is used to predict the pixel prediction value.

10. The training method according to claim 9, wherein, The step of extracting feature information of the pixel block using the machine learning model based on the first embedded feature tensor includes: The pixels in the image are positionally encoded to generate a positionally encoded tensor; A second embedding feature tensor is generated based on the position encoding tensor and the first embedding feature tensor; Based on the second embedded feature tensor, the machine learning model is used to extract feature information of pixels in the image.

11. The training method according to claim 10, wherein, Position encoding is performed on the first embedded feature tensor to generate a position encoded tensor, including: Based on different directions, the first embedded feature tensor is divided into multiple embedded feature sub-tensors; Position encoding is performed on the multiple embedded feature sub-tensors respectively to generate multiple position-encoded tensors.

12. The training method according to claim 1, wherein, Expanding at least one pixel of the image sample to be processed into a pixel block includes: The image sample to be processed is divided into multiple sub-images; Expand at least one pixel of the plurality of sub-images into a pixel block.

13. The training method according to claim 12, wherein, The step of dividing the image sample to be processed into multiple sub-images includes: The image to be processed is divided into multiple image blocks; For different image channels, each of the multiple image blocks is divided into multiple sub-images.

14. The training method according to any one of claims 1-13, wherein, The first mask is the same for processing different pixel blocks, and all mask values ​​in the first mask are the same, or The first mask is the same for processing different pixel blocks, but the mask values ​​in the first mask are all different, or The first mask for processing different pixel blocks is different, and the mask values ​​in each first mask are the same, or The first mask for processing different pixel blocks is different, and the mask values ​​in each first mask are different from each other.

15. An image processing method, comprising: At least one pixel of the image sample to be processed is expanded into a pixel block, the pixel block comprising multiple pixels to be predicted; The pixel block is processed using the first mask to obtain the mask processing result; Based on the mask processing result, the machine learning model is used to predict the pixel prediction values ​​of multiple pixels to be predicted in the pixel block, thereby obtaining the processed image sample. The machine learning model is trained using the training method described in any one of claims 1 to 14.

16. A training device for a machine learning model, comprising: An expansion unit is used to expand at least one pixel of an image sample to be processed into a pixel block, the pixel block including multiple pixels to be predicted, and to process the pixel block using a first mask to obtain a mask processing result. The prediction unit is used to predict the pixel prediction values ​​of multiple pixels to be predicted in the pixel block based on the mask processing result and the machine learning model, so as to obtain the processed image sample. The training unit is used to train the machine learning model based on the pixel prediction values ​​and pixel annotation values ​​of the plurality of pixels to be predicted. Wherein, the at least one pixel is a seed pixel, and the expansion unit assigns a corresponding initial value to each of the plurality of pixels to be predicted based on the arbitrary position of the seed pixel in the pixel block in the first mask and the relationship between the mask values ​​in the first mask, so as to obtain the plurality of pixels to be predicted with the initial value as the mask processing result. The prediction unit uses the machine learning model to predict the pixel prediction value of the plurality of pixels to be predicted based on the initial value corresponding to each pixel to be predicted.

17. An image processing apparatus, comprising: An expansion unit is used to expand at least one pixel of an image sample to be processed into a pixel block, the pixel block including multiple pixels to be predicted, and to process the pixel block using a first mask to obtain a mask processing result. The prediction unit is used to predict the pixel prediction values ​​of multiple pixels to be predicted in the pixel block based on the mask processing result and using the machine learning model, so as to obtain the processed image sample. The machine learning model is trained using the training method according to any one of claims 1 to 14.

18. An electronic device comprising: Memory; and A processor coupled to the memory, the processor being configured to execute, based on instructions stored in the memory, a training method for a machine learning model according to any one of claims 1 to 14, or an image processing method according to claim 14.

19. A non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a machine learning model according to any one of claims 1 to 14, or an image processing method according to claim 14.

Images