A method of image precoding based on attention mechanism

Abstract

The present invention belongs to the field of image coding technology, and specifically discloses a method for image precoding based on an attention mechanism. The present invention proposes a scenario in which image transmission is required after image instance segmentation is performed, and the image is firstly subjected to an attention mechanism operation based on the result obtained by image segmentation, and then the image is encoded, which can not only ensure the image quality and details of the ROI area, but also remove the redundant information of the non-ROI area of ​​the image, and the present invention can also flexibly configure the weight coefficient, so as to flexibly adjust between detail retention and redundancy removal, so as to balance the compression ratio and quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of image coding, and in particular relates to a method for image precoding based on an attention mechanism. Background Art

[0002] Segmentation algorithms have been increasingly used in areas such as guidance devices, autonomous driving, industrial quality inspection, and medical image analysis, and these scenarios require algorithms to respond in real time. With the rapid growth of computing power, more and more end-side devices can directly implement segmentation algorithm reasoning on the device side. End-side device computing can process data instantly without waiting for cloud response, which not only significantly reduces the time delay of data transmission and processing, but also maintains a certain processing capacity in the case of poor network conditions or interruptions, ensuring the continuity and stability of segmentation algorithm reasoning.

[0003] When the end-side device completes the calculation, the original image needs to be sent to the data center for subsequent retrieval, analysis, and backup of the data, as well as updating of the algorithm based on the data. For images that have already executed the segmentation algorithm, the parts of the image that need special attention are already known, that is, the areas that have been segmented in the image. However, the current existing image coding either uses lossless compression to retain the overall quality and details of the image, but this method will increase the hardware power consumption and time of encoding, and cannot remove redundant information in non-ROI areas of the image; or uses lossy compression to reduce the overall quality and details of the image, but this method cannot retain the details and quality of the image ROI area. In addition, the existing lossy or lossless coding is fixed, and cannot be flexibly adjusted between retaining details and removing redundancy, making it difficult to balance compression ratio and quality. Summary of the invention

[0004] In order to solve the problems mentioned in the above background technology, the present invention proposes an image precoding method based on attention mechanism.

[0005] A method for image precoding based on attention mechanism, the steps are as follows:

[0006] Step S1: The end-side device loads the original image and uses the trained image segmentor to perform instance segmentation on the original image.

[0007] Specifically, the image segmenter is an artificial neural network that outputs detection boxes and masks at the same time. The calculation formula of the image segmenter is:

[0008] ;

[0009] ;

[0010] ;

[0011] In the formula, To increase the artificial neural network of detection box output and mask output, is the original image, is the pixel coordinate in the original image, for The extracted features, is the fully connected layer, is a binary classification function, is the detection box output of the artificial neural network, is the mask output by the artificial neural network, is the pixel coordinate in the mask.

[0012] After the end-side device completes instance segmentation, it is also necessary to send the original image to the data center for subsequent retrieval, analysis and backup of the data, as well as updating the algorithm based on the data. Generally, the traditional approach is to directly encode the image and then send the encoded data. However, since instance segmentation has been performed on the image, that is, the ROI area in the image is already known, if the image is still encoded as a whole, it will not only increase the hardware power consumption and time of encoding, but also increase the bandwidth overhead of data transmission after encoding, and affect the transmission rate of the encoded data. Based on this. The present invention proposes to perform an attention mechanism operation on the image on the basis of instance segmentation that has been completed on the end side, thereby ensuring the quality of the ROI area in the image while reducing the quality of the non-ROI area in the image, thereby reducing the hardware power consumption and time during image encoding, and reducing the bandwidth overhead of data transmission after network coding.

[0013] Step S2, obtaining the output result of the image segmenter, and obtaining the mask image from the output result, the specific steps are as follows:

[0014] First, get the height and width of the original image; then, resize the mask to the height and width of the original image; finally, merge the output masks into a single mask image.

[0015] This step is to obtain the ROI area and non-ROI area in the original image from the output result of the image segmenter, upsample the mask image to the size of the original image, and merge multiple ROI areas in the output mask into one image of the original image.

[0016] Step S3: configure the weight of the attention mechanism according to the obtained mask image. The specific steps are as follows:

[0017] Step S301, configure the weight coefficient of the attention mechanism, the specific constraint formula is:

[0018] ;

[0019] In the formula, Represents the weight coefficient of the ROI area, Represents the weight coefficient of the non-ROI area;

[0020] In different application scenarios, there are different backup, usage and display requirements for the original image of the acquired image ROI area. By setting the size of the ROI area weight coefficient and the non-ROI area weight coefficient, not only can the quantitative compression degree of the original image ROI area and non-ROI area be flexibly set, but also the on-demand configuration between the overall image quality and the image file size can be achieved, thereby meeting different usage scenarios.

[0021] Step S302: construct a weight coefficient matrix according to the weight coefficients. The calculation formula of the weight coefficient matrix is:

[0022] ;

[0023] In the formula, is the merged mask image, is the pixel coordinate in the merged mask image.

[0024] Step S4, performing an attention mechanism operation on the original image according to the configured attention mechanism weights. Specifically, the attention mechanism operation includes a color space weighted calculation and a frequency component weighted calculation, wherein the formula for the color space weighted calculation is:

[0025] ;

[0026] In the formula, is the image obtained after performing color space weighted calculation, are the coordinates in the image obtained after weighted calculation in color space, For pixel-by-pixel multiplication, To round down, is the original image, is the merged mask image, is the pixel coordinate in the merged mask image.

[0027] The formula for calculating the frequency component weighting is:

[0028] ;

[0029] ;

[0030] In the formula, is the quantized discrete cosine transform coefficient matrix, is the value in the discrete cosine transform coefficient matrix after quantization, N is the number of blocks divided into discrete cosine transforms, is the original image, is the cosine function, represents pi, is the normalization coefficient, and are the coefficient values ​​corresponding to the horizontal and vertical coordinates in the normalized coefficients, To round down, is the merged mask image, is the pixel coordinate in the merged mask image, is the discrete cosine transform coefficient matrix.

[0031] For the scenario where the image needs to be transmitted after the image is segmented, the current existing image coding either uses lossless compression to retain the overall quality and details of the image; or uses lossy compression to reduce the overall quality and details of the image, which cannot take into account both detail retention and redundancy removal. The present invention proposes to first perform an attention mechanism operation on the image, that is, by giving more weight to the ROI area and less weight to the non-ROI area, and then losslessly encode the image, which can not only ensure the image quality and details of the ROI area, but also remove the redundant information of the non-ROI area of ​​the image. In addition, the weight coefficient can be flexibly configured to flexibly adjust between detail retention and redundancy removal, thereby balancing the compression ratio and quality.

[0032] Step S5, encoding the result obtained after the attention mechanism operation, specifically includes the following steps:

[0033] First, the discrete cosine transform coefficient matrix after statistical quantization The frequency of occurrence of each coefficient in ;

[0034] Then, a Huffman tree is constructed based on the statistical frequency; then, the Huffman codeword is generated by the constructed Huffman tree; finally, the image is obtained by replacing the generated Huffman codeword with the color space weighted calculation. The pixels in .

[0035] After obtaining the result of the attention mechanism operation, by encoding the result obtained after the attention mechanism operation, the result of the attention mechanism operation can be used to perform lossless encoding on the ROI area in the image, and perform quantization compression of the color space and quantization compression of the frequency component on the non-ROI area in the image, thereby ensuring the quality of the ROI area in the image on the basis of achieving quantization compression of the non-ROI area in the image.

[0036] Step S6: The terminal device transmits the encoded data.

[0037] The present invention proposes a method for image precoding based on attention mechanism, which has the following beneficial effects compared with the existing technology:

[0038] The present invention proposes a scenario in which image transmission is required after image instance segmentation is performed. First, an attention mechanism operation is performed on the image based on the result of image segmentation, and then the image is encoded. This method can not only ensure the image quality and details of the ROI area, but also remove redundant information in the non-ROI area of ​​the image. Moreover, the present invention can also flexibly configure weight coefficients, so that flexible adjustment can be made between detail retention and redundancy removal, thereby balancing the compression ratio and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a method for image precoding based on an attention mechanism of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose and features of the present invention more obvious and easy to understand, the present technical solution is described in detail below through embodiments and in conjunction with the accompanying drawings.

[0041] like Figure 1 As shown in FIG. 1 , a method for image precoding based on an attention mechanism is implemented as follows:

[0042] Step S1: The end-side device loads the original image and uses the trained image segmentor to perform instance segmentation on the original image.

[0043] Specifically, the image segmenter is an artificial neural network that outputs detection boxes and masks at the same time. The calculation formula of the image segmenter is:

[0044] ;

[0045] ;

[0046] ;

[0047] In the formula, To increase the artificial neural network of detection box output and mask output, is the original image, is the pixel coordinate in the original image, for The extracted features, is the fully connected layer, is a binary classification function, is the detection box output of the artificial neural network, is the mask output by the artificial neural network, is the pixel coordinate in the mask.

[0048] Optionally, the mask here can be a binary image in which background pixels are usually assigned a value of 0, or can be a grayscale image in which background pixels are usually assigned a value of 0.

[0049] Step S2, obtaining the output result of the image segmenter, and obtaining the mask image from the output result, the specific steps are as follows:

[0050] First, get the height and width of the original image; then, adjust the size of the mask to the height and width of the original image;

[0051] Optionally, the mask may be adjusted to the height and width of the original image by using nearest neighbor interpolation or bilinear interpolation, and the size of the post-processed mask is consistent with the original image.

[0052] Finally, the output masks are merged into a single mask image.

[0053] In the specific implementation process, if there are multiple instance segmentation targets in the combined mask image, multiple to merge.

[0054] Step S3: configure the weight of the attention mechanism according to the obtained mask image. The specific steps are as follows:

[0055] Step S301, configure the weight coefficient of the attention mechanism, the specific constraint formula is:

[0056] ;

[0057] In the formula, Represents the weight coefficient of the ROI area, Indicates the weight coefficient of the non-ROI area.

[0058] During the specific implementation process, different ROI area weight coefficients and non-ROI area weight coefficients may be set according to the actual computing power of the end-side device, the requirements for data backup, and the bandwidth for data transmission.

[0059] Step S302: construct a weight coefficient matrix according to the weight coefficients. The calculation formula of the weight coefficient matrix is:

[0060] ;

[0061] In the formula, is the merged mask image, is the pixel coordinate in the merged mask image.

[0062] In the specific implementation process, for each pixel point (i, j) in the ROI area, let its value be equal to the weight coefficient of the ROI area; for each pixel point (i, j) in the non-ROI area, let its value be equal to the weight coefficient of the non-ROI area, thus the weight matrix can be obtained .

[0063] Step S4, performing an attention mechanism operation on the original image according to the configured attention mechanism weights. Specifically, the attention mechanism operation includes a color space weighted calculation and a frequency component weighted calculation, wherein the formula for the color space weighted calculation is:

[0064] ;

[0065] In the formula, is the image obtained after performing color space weighted calculation, are the coordinates in the image obtained after weighted calculation in color space, For pixel-by-pixel multiplication, To round down, is the original image, is the merged mask image, is the pixel coordinate in the merged mask image.

[0066] The formula for calculating the frequency component weighting is:

[0067] ;

[0068] ;

[0069] In the formula, is the quantized discrete cosine transform coefficient matrix, is the value in the discrete cosine transform coefficient matrix after quantization, N is the number of blocks divided into discrete cosine transforms, is the original image, is the cosine function, represents pi, is the normalization coefficient, and are the coefficient values ​​corresponding to the horizontal and vertical coordinates in the normalized coefficients, To round down, is the merged mask image, is the pixel coordinate in the merged mask image, is the discrete cosine transform coefficient matrix.

[0070] In the specific implementation process, it is necessary to first perform discrete cosine transform on the original image, and then perform frequency component weighted calculation on the image after discrete cosine transform.

[0071] Step S5, encode the result obtained after the attention mechanism operation, and the implementation steps are as follows:

[0072] First, the discrete cosine transform coefficient matrix after statistical quantization The frequency of occurrence of each coefficient in ; then, a Huffman tree is constructed based on the statistical frequency; then, the Huffman codeword is generated by the constructed Huffman tree; finally, the image is obtained by replacing the generated Huffman codeword with the color space weighted calculation. The pixels in .

[0073] Optionally, arithmetic coding may be used to encode the image obtained after performing the color space weighted calculation using the quantized discrete cosine transform coefficient matrix.

[0074] Step S6: The terminal device transmits the encoded data.

[0075] So far, according to the method disclosed in the present invention, the working process of the present invention has been implemented once.

[0076] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

[0077] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

Claims

1. A method for image precoding based on attention mechanism, characterized in that: Includes steps: Step S1: The end-side device loads the original image and uses the trained image segmentor to perform instance segmentation on the original image. The image segmenter is an artificial neural network that outputs a detection frame and a mask at the same time. The calculation formula of the image segmenter is: ； ； ； In the formula, To increase the artificial neural network of detection box output and mask output, is the original image, is the pixel coordinate in the original image, for The extracted features, is the fully connected layer, is a binary classification function, is the detection box output of the artificial neural network, is the mask output by the artificial neural network, is the pixel coordinate in the mask; Step S2, obtaining the output result of the image segmenter, and obtaining a mask image from the output result; Step S3, configuring the weight of the attention mechanism according to the obtained mask image; Step S3 includes the steps of: Step S301, configure the weight coefficient of the attention mechanism, the specific constraint formula is: ； In the formula, Represents the weight coefficient of the ROI area, Represents the weight coefficient of the non-ROI area; Step S302: construct a weight coefficient matrix according to the weight coefficients. The calculation formula of the weight coefficient matrix is: ； In the formula, is the merged mask image, is the pixel coordinate in the merged mask image; Step S4, performing an attention mechanism operation on the original image according to the configured attention mechanism weight; The attention mechanism operation includes color space weighted calculation, and the formula of color space weighted calculation is: ； In the formula, is the image obtained after performing color space weighted calculation, are the coordinates in the image obtained after weighted calculation in color space, For pixel-by-pixel multiplication, To round down, is the original image, is the merged mask image, is the pixel coordinate in the merged mask image; The attention mechanism operation also includes a frequency component weighted calculation, and the formula for the frequency component weighted calculation is: ； ； In the formula, is the quantized discrete cosine transform coefficient matrix, is the value in the discrete cosine transform coefficient matrix after quantization, N is the number of blocks divided into discrete cosine transforms, is the original image, is the cosine function, represents pi, is the normalization coefficient, and are the coefficient values ​​corresponding to the horizontal and vertical coordinates in the normalized coefficients, To round down, is the merged mask image, is the pixel coordinate in the merged mask image, is the discrete cosine transform coefficient matrix; Step S5, encoding the result obtained after the attention mechanism operation; Step S6: The terminal device transmits the encoded data.

2. The method for image precoding based on attention mechanism according to claim 1, characterized in that: Step S2 includes: Step S201, obtaining the height and width of the original image; Step S202, adjusting the size of the mask to the height and width of the original image; Step S203: merging the output masks into a single mask image.

3. The method for image precoding based on attention mechanism according to claim 1, characterized in that: Step S5 comprises the steps of: Step S501: Count the discrete cosine transform coefficient matrix after quantization The frequency of occurrence of each coefficient in ; Step S502, constructing a Huffman tree according to the statistical frequency; Step S503, generating a Huffman code word by using the constructed Huffman tree; Step S504: Replace the generated Huffman codewords with the color space weighted calculation to obtain the image The pixels in .

Images