Image processing method, system, encoder and computer-readable storage medium

Abstract

The present disclosure relates to an image processing method and system, an encoder, a computer-readable storage medium, and a computer technical field. The method of the present disclosure is, for the feature matrix of various channels of an image output by an intermediate layer of a neural network, the intermediate layer is a layer between an input layer and an output layer, and determines rows and columns having the same feature value as rows and columns to be compressed at the edge of the feature matrix, deletes the feature values ​​of the rows and columns to be compressed and reserves the remaining feature values ​​as reserved values ​​of the channel feature matrix, compresses the feature values ​​of the rows and columns to be compressed, obtains edge values ​​of the channel feature matrix, and transmits them to a decoder after encoding the reserved values ​​and edge values ​​of the channel feature values ​​and the amount of the rows and columns to be compressed, and the decoder recovers the channel feature matrix.

Description

[Technical field]

[0001] This disclosure is based on and claims priority to China Invention Application No. 202110339327.1, filed on March 30, 2021, the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to the field of computer technology, and in particular to image processing methods, systems, encoders, and computer-readable storage media. [Background technology]

[0003] The Video Coding for Machines (VCM) Expert Group is a group of experts who are working on the development of a method for coding compressed video or feature code streams extracted from video. Encoding The aim is to standardize image processing technology that can be used for a variety of machine tasks (mechanization tasks) while retaining high compression efficiency and intelligent task performance, and is useful for machine vision (mechanization outlook) and human-machine hybrid vision applications. Summary of the Invention [Problem to be solved by the invention]

[0004] Deep Learning Capabilities for Machine Vision Encoding So, Encoding The layer of the neural network in which the object of interest is located can first be determined, and feature codes are output from the determined layer of the neural network, which are then used in various subsequent video processing tasks. [Means for solving the problem]

[0005] Deep Learning for Machine Vision Features A commonly used neural network for encoding is the convolutional neural network. (padding)The operation is often introduced before the convolution of a convolutional neural network to add some pixels around the image and set these pixels to 0, such as Pre-configured Initialize to a value.

[0006] According to an embodiment of the present invention, there is provided an image processing method for a feature matrix of each channel of an image output from an intermediate layer of a neural network, the image processing method comprising the steps of: At the edge of the feature matrix One or more rows with the same feature value and One or more columns with the same feature value and of , pressure Determine one or more rows to shrink and one or more columns to compress determining one or more rows having the same feature value and one or more columns having the same feature value, the intermediate layer being a layer between the input layer and the output layer; , delete the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed, and The above The feature matrix of each channel is reserved as a reserved value. To do compressing the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed to obtain one or more edge values ​​of the feature matrix for each channel; To do, the reserved values ​​and the one or more edge values ​​of the feature matrix for each channel, the number of rows to be compressed and the number of columns to be compressed Encode , the above Encoding The result of the above is sent to a decoder, and the decoder calculates the feature matrix of each channel. and restoring .

[0007] In some embodiments, one or more rows having the same feature value at an edge of the feature matrix. Same as One or more columns with the same feature values of decision Things to do is the feature matrix of the channel of the image output from the intermediate layer of the neural network. 、 Select as the feature matrix to be detected To do , downward from the first row of the feature matrix to be detected, and upward from the last row of the feature matrix to be detected. To , respectively, of the feature matrix to be detected Should be compressedDetect one or more rows having the same feature value as one or more rows, and search the feature matrix from the first column backwards and from the last column forwards. Special The above mentioned characteristic matrix Should be compressed Detect one or more columns with the same feature value as one or more columns To do The detection target Special It is in the same position as the one or more rows to be compressed of the characteristic matrix. The above For each channel The above determining one or more rows of a feature matrix as the one or more rows of the feature matrix of each channel to be compressed; The above One or more columns of the feature matrix for each channel that are in the same position as one or more columns to be compressed are Pressure Determine one or more columns to be compressed. and, of include .

[0008] In some embodiments, The above should be compressed Number of rows greater than 1 and The above should be compressed the number of columns of the feature matrix for each of the channels, in addition to the number of columns of the feature matrix for each of the channels, Encoding performing the above step of: and before Edge values ​​above 1 are compressed in a predefined manner To do and before The feature matrix of each channel The above Compressed reserved values ​​and The above Compressed edge value greater than or equal to 1 and encoding a number of the one or more rows to be compressed and a number of the one or more columns to be compressed. .

[0009] In some embodiments, the retention values ​​and the one or more edge values ​​of the feature matrix for each of the channels are In the above-mentioned preset method compression do For the feature matrix of each channel, the reserved value of the feature matrix of each channel is and before Edge values ​​and prior To The feature matrix of each channel is corresponding to a set bit depth value. Order of magnitude index decision To do , The aboveThe reserved value and the one or more edge values ​​of the feature matrix of each channel are The size index According to quantum The feature matrix of each channel is The above the compressed reserve value and the compressed one or more edge values ,quantum transformation was done Reserved Value and Quantized 1 or more NoE Get the edge value Including .

[0010] In some embodiments, The above of the feature matrix to be detected First Down the row To , and upwards from the last row of the feature matrix to be detected To ,each 、 One or more rows with the same feature value of detection Things to do (1) The above For the feature matrix to be detected, one or more top Initialize the number of rows to zero. To do (2) Determine whether the feature values ​​in the first row are the same, and if so, The above Compression Ki 1 The above top Increment the number of lines by one and continue with the next line as the current line To do , (3 ) current Determine whether the feature value of the current row is the same as that of the previous row, and if so, compress the compressed Should 1 The above top Set the number of rows to 1. t Increment and continue with the next line as the current line To do , (4 ) current Current practice Special The feature value is different from the previous row. in( 3) Steps Repeatedly compress Ki 1 The above top Row and The above Should be compressed top Get the number of rows and and To be detected Special Conscription Vertical To Obtained by mirroring matrix Check Should be issued New Feature matrices (1) to (4) Steps Go to one or more bottom rows to be compressed and a number of said one or more bottom rows to be compressed; 1 or more to be compressed top Row and The above 1 or more to be compressed top Number of rows and Get and .

[0011] In some embodiments, The above To be detected Special From the first row of the conscription procession to the back and , and The above To be detected Special From the last row of the conscription procession to the front and , each, One or more columns with the same feature value of Detect thing teeth , the above To be detected Special Conscription Turn The matrix obtained by substituting Perform steps (1) to (4) on the new feature matrix to be detected. 1 or more left columns to compress and The one or more left columns to be compressed Number and Used as obtaining one or more top rows to compress and a number of the one or more top rows to compress; To be detected Special Conscription The matrix obtained by vertically mirroring the transposed matrix is Should be detected New As a feature matrix Follow steps (1) to (4) One or more right columns to be compressed and one or more right columns to be compressed Number and Used as obtaining one or more top rows to compress and a number of the one or more top rows to compress. .

[0012] In some embodiments, the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed are compressed to obtain one or more edge values ​​of the feature matrix for each of the channels. of compression Things to do is a feature value of the one or more rows to be compressed and the one or more columns to be compressed. Each of the above As the one or more edge values ​​of the feature matrix of the channel First It is possible to express it as a value.

[0013] In some embodiments, The above Encode D is entropy encoder D It is.

[0014] In some embodiments, The method comprises: before By the above decoder, One or more rows to be compressed Number of and one or more of the components to be compressed. Number of columns In addition ,before The feature matrix of each channel said compressed reserved value; The compressed 1 or more To get the edge value To, Decode the received data To do , the decoder The feature matrix Compressed reserved values and before The compressed edge value is and Unzip the stomach , obtaining the decompressed retention value of the feature matrix for each of the channels and one or more decompressed edge values. To do, For the feature matrix for each channel, the decoder The above For each channel The above To get the feature matrix 、 The number of one or more rows to be compressed; The above The number of columns (one or more) to be compressed. depending on , The above For each channel The above Feature matrix The above Around the decompressed reserved value, The above Corresponding to the decompressed reserved value The above Unpacked 1 or more Edge value and further comprising padding. .

[0015] In some embodiments, further comprising: index The feature matrix of each channel The above Reserved Value and the above Edge value greater than or equal to 1 and in advance setting was done Determined according to the bit depth value Includes size indicators, The decoder and For each channel The above Feature matrix The above Compressed reserved values and The compressed 1 or more Edge Value of Unzip Things to do For the feature matrix of each channel, the decoder generates a feature matrix corresponding to the feature matrix of each channel. The above Size index Then, the compressed reserved values ​​and the previous Recording Shrunken 1 or more Edge Value and , respectively, to obtain the decompressed reserved values ​​and the previous Notes Frozen 1 or more To get the edge value include .

[0016] According to another embodiment of the present disclosure, an encoder is provided, the encoder including: For , At the edge of the feature matrix One or more rows having the same feature value and one or more columns having the same feature value are determined as one or more rows to be compressed and one or more columns to be compressed at the edges of the feature matrix. do Module for determining which information to compress The intermediate layer is a layer between the input layer and the output layer. For the feature matrix of each channel, the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed are deleted, and the remaining feature values ​​are The above The feature matrix of each channel is reserved as a reserved value. do a retention value determination module, for the feature matrix of each of the channels, determining the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed to obtain one or more edge values ​​of the feature matrix of each of the channels; of compression do an edge value compression module; and a filter for filtering the reserved values ​​of the feature matrix for each of the channels and the one or more edge values. 、 and the number of one or more rows to be compressed and the number of one or more columns to be compressed. Encode the , the above Encoding The result of the above is sent to a decoder, and the decoder outputs the feature matrix Encoding to restore A module.

[0017] According to a further embodiment of the present disclosure, there is provided an encoder, the encoder comprising: a processor; and a memory coupled to the processor for storing instructions. A memory , The command is When executed by the processor, the image processing method described in the previous embodiment is performed by the processor. Let R 、 A memory is provided.

[0018] According to another embodiment of the present disclosure, Non-transient A computer-readable storage medium is provided, Non-transient A computer-readable storage medium is provided for storing a computer program. Non-transient A computer-readable storage medium, the computer program being adapted, when executed by a processor, to perform the steps of the image processing method described in the previous embodiments. Let do.

[0019] According to another embodiment of the present disclosure, there is provided an image processing system comprising an encoder and a decoder as described in the previous embodiment, wherein the decoder reconstructs the feature matrix for each of the channels according to received data transmitted by the encoder.

[0020] In some embodiments, the decoder further comprises: Number of and one or more of the components to be compressed. Number of columns In addition, The feature matrix of each channel The compressed reserve value and the compressed 1 or more To get the edge value, The above Received Data of Decoding death , each of the channels The feature matrix Compressed reserved values and before The compressed one or more edge values and Unzip the stomach , the decompressed retention values ​​of the feature matrix for each of the channels and the decompressed one or more edge values and and for the feature matrix of each channel, The above For each channel The above the number of one or more rows to be compressed to obtain a feature matrix; The above The number of columns (one or more) to be compressed. depending on , The above For each channel The above Feature matrix The above Around the decompressed reserved value, The above Corresponding to the decompressed reserved value The above Unpacked 1 or more Edge value Padding do.

[0021] In some embodiments, the received data further comprises a feature matrix corresponding to each of the channels. Size index The reserved value of the feature matrix for each channel and Edge value greater than or equal to 1 and in advance setting was done Determined according to the bit depth value Size index ,of Including The decoder generates a feature matrix corresponding to the feature matrix of each channel. Before Note Size index the compressed reserved values ​​of the feature matrix for each channel and Compressed Edge value greater than or equal to 1 and and multiplying the decompressed reserved values ​​of the feature matrix for each channel by Unpacked 1 or more NoE Get the offset value.

[0022] Other features and advantages of the present invention will be made apparent from the following detailed description of illustrative embodiments thereof, which proceeds with reference to the accompanying drawings.

[0023] The accompanying drawings are configured to provide a further understanding of the present disclosure, are incorporated in and constitute a part of this specification, illustrate embodiments of the present invention, and, together with the exemplary embodiments of the present application, serve to explain the present disclosure but are not intended to limit the disclosure. [Brief description of the drawings]

[0024] [Figure 1]1 shows a flowchart of an image processing method according to some embodiments of the present disclosure. [Diagram 2] 13 shows a flowchart of an image processing method according to another embodiment of the present disclosure. [Diagram 3] 10 is a flowchart of an image processing method according to yet another embodiment of the present invention. [Figure 4] 1 shows a structural diagram of an encoder according to some embodiments of the present disclosure. [Diagram 5] FIG. 11 is a structural diagram of an encoder according to another embodiment of the present invention. [Figure 6] FIG. 13 is a structural diagram of an encoder according to still another embodiment of the present invention. [Figure 7] 1 shows a structural diagram of an image processing system according to some embodiments of the present disclosure.

[0025] With reference to the figures of the embodiments, the technical solutions of the embodiments of the present disclosure are described below clearly and completely. It is obvious that the present disclosure is not all the embodiments, and only some of the embodiments are described herein. The following description of at least one exemplary embodiment is merely exemplary in nature, and is not intended to limit the present invention, its application, or use in any way. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts are included within the protection scope of the present disclosure.

[0026] What the inventors discovered is that deep learning feature encoding for machine vision is a promising approach to computational modeling in convolutional neural networks. Padding By this operation, the feature code of the image output from the intermediate layer includes the edge Padding Many features are generated and processed at the edge. Padding However, transmitting all these calculated feature values ​​to the decoder may result in wasted bandwidth and reduced transmission efficiency.

[0027] In this regard, the technical problem solved by the present disclosure is how to improve transmission efficiency to a decoder and reduce bandwidth waste in deep learning feature encoding for machine vision.

[0028] According to this disclosure, deep learning features for machine vision encoding In this case, the encoder extracts the feature code of the video image and Decoding and send it to the decoder for further processing. The encoder and decoder can be deployed separately, with the encoder deploying a part of the neural network structure and the decoder deploying another part of the neural network structure. Based on the computational capabilities of the encoder, a part of the neural network structure can be selectively deployed in the encoder. For example, a certain hidden layer and the layers before it of the neural network are placed in the encoder, and the layers after a certain hidden layer are placed in the decoder. The encoder sends the features output by a particular hidden layer to the decoder for further processing. do The present invention proposes an image processing method for solving the problems of bandwidth waste and reduced transmission efficiency caused by transmitting a large amount of data from an encoder to a decoder, which will be described with reference to Figs.

[0029] 1 is a schematic diagram of an image processing method according to some embodiments of the present disclosure. As shown in FIG 1, the method of these embodiments includes steps S102 to S108.

[0030] In step S102, the feature matrix of each channel of the image output from the intermediate layer of the neural network is Edge One or more rows having the same feature and one or more columns having the same feature are determined as one or more rows to be compressed and one or more columns to be compressed.

[0031] The inventors realized that PaddingAfter the operation, the edge rows and columns of the image feature matrix have particularly similar feature values, or multiple rows and columns have the same feature values. Therefore, the image feature matrix can be compressed. The intermediate layer is a layer between the input layer and the output layer. Deep Learning Features for Machine Vision encoding In this case, the intermediate layer may be the last layer of a neural network deployed in the encoder, and may be selected based on the actual processing capabilities of the encoder or other components, but is not limited to this particular disclosure.

[0032] In some embodiments, a feature matrix of a channel of an image output from an intermediate layer of a neural network is selected as a feature matrix to be detected, one or more rows having the same feature value are detected downward from the first row of the feature matrix to be detected and upward from the last row of the feature matrix to be detected as rows to be compressed of the feature matrix, one or more columns having the same feature value are detected backward from the first column of the feature matrix to be detected and forward from the last column of the feature matrix to be detected as one or more columns to be compressed of the feature matrix to be detected, one or more rows of the feature matrix of each channel that are in the same positions as the one or more rows to be compressed of the feature matrix to be detected are determined as one or more rows to be compressed of the feature matrix of each channel, and one or more columns of the feature matrix of each channel that are in the same positions as the one or more columns to be compressed of the feature matrix to be detected are determined as one or more columns to be compressed of the feature matrix of each channel. The feature matrix of any channel can be selected as the feature matrix to be detected.

[0033] Several embodiments of methods for determining one or more rows to compress downward from the first row of a feature matrix to be detected are described below with reference to FIG.

[0034] In step S202, for the feature matrix to be detected, one or more top Row number sameTopCnt is initialized to 0.

[0035] In step S204 First rowIf they are the same, the process proceeds to step S206, and if they are different, the process proceeds to step S212.

[0036] In step S206, sameTopCnt is incremented by one and the method proceeds to step S208 with the next row as the current row.

[0037] In step S208, it is determined whether the feature amount of the current line is the same as that of the previous line. If they are the same, the process proceeds to step S210, and if they are different, the process proceeds to step S212.

[0038] In step S210 sameTopCnt is incremented by one and the method repeats step S208 with the next row as the current row.

[0039] In step S212, one or more top A row and one or more rows to compress. top The number of rows sameTopCnt is determined.

[0040] In some embodiments, the method of detecting one or more rows of a feature matrix to be detected that have the same feature value from the last row upwards includes: in the vertical direction The matrix mirrored in the above is used as the new feature matrix to be detected. do , one or more to be compressed by executing steps S202 to S208. top Rows and one or more to be compressed top It provides a way to get the number of rows that should be compressed. bottom Rows and one or more to be compressed bottom The number of lines is used as sameBotCnt.

[0041] In some embodiments, the method for detecting one or more columns of a feature matrix to be detected that have the same feature value backward from the first column includes the step of: of New detection target feature matrix As Execute steps S202 to S208 to obtain one or more top Rows and one or more to be compressed top obtaining a number of rows, which are one or more left columns to be compressed and one or more left columns to be compressed; number s Used as ameLeftCnt.

[0042] In some embodiments, the method for detecting one or more columns of a feature matrix to be detected that have the same feature value from the last column forward is to calculate a transpose of the feature matrix to be detected. Vertical Mirrored matrix in the direction of New feature matrix to be detected As , one or more to be compressed by executing steps S202 to S208. top Rows and one or more to be compressed top obtaining a number of rows, which are one or more right columns to be compressed and one or more right columns to be compressed; number s Used as ameRightCnt.

[0043] In step S104, for each channel feature matrix, a 1 to be compressed is The above Rows and 1 to be compressed More columns The features of are removed and the remaining features are stored as reserved values ​​of the feature matrix of that channel.

[0044] For example, a cut operation is performed on features at the edge of feature matrix F (H*W, e.g., H rows and W columns) to remove the top row of sameTopCnt, the bottom row of sameBotCnt, the left column of sameLeftCnt, and the right column of sameRightCnt while leaving feature values ​​F' in the range of (H-sameTopCnt-sameBotCnt)*(W-sameLeftCnt-sameRightCnt) as reserved values ​​in the feature matrix.

[0045] In step S106, for the feature matrix of each channel, feature values ​​of one or more rows to be compressed and one or more columns to be compressed are compressed to obtain one or more edge values ​​of the feature matrix of that channel.

[0046] In some embodiments, the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed are represented by values ​​that act as one or more edge values ​​of the feature matrix for each channel. If the one or more top rows to be compressed, the one or more bottom rows to be compressed, the one or more left columns to be compressed, and the one or more right columns to be compressed correspond to different feature values, then the one or more top rows to be compressed are represented by values, the one or more bottom rows to be compressed are represented by values, and the one or more left columns to be compressed are represented by values; Should be compressed The right column, 1 or more, is represented by a value.

[0047] In step S108, the number of rows to be compressed (one or more) and the number of columns to be compressed (one or more) are ,and The reserved values ​​of the feature matrix for each channel and edge values ​​of 1 or more are Encoding The decoder then calculates the feature matrix for each channel. Decoding The encoded data is then sent to the decoder for decoding.

[0048] For example, the feature matrix for channel i is

number

number

number

number

number

number

number

[0049] Reserved values ​​of the feature matrix for each channel and 1 or more edge values ,and The number of one or more rows to be compressed and the number of one or more columns to be compressed teeth , to form a bitstream Encoding The entropy may be the calculated entropy, which is transmitted to the decoder.

[0050] In the method of the above embodiment, for the feature matrix of each channel of the image output from the intermediate layer of the neural network, in One or more rows having the same feature and one or more columns having the same feature are determined as one or more rows to be compressed and one or more columns to be compressed, and to obtain a reserved value of the feature matrix of each channel, the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed are deleted, and the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed are compressed to obtain one or more edge values ​​of the feature matrix of each channel, and in addition to the number of one or more rows to be compressed and the number of one or more columns to be compressed, the reserved value and one or more edge values ​​of the feature matrix of each channel are Encoding The information is then processed and transmitted to the decoder. The decoder can restore the feature matrix for each channel based on the received information. The method of the above embodiment can improve the compression rate of the features output by the intermediate layer of the neural network. encodingIn, the feature matrix of each channel output by the intermediate layer is compressed by the method of the above embodiment and can be transmitted to the decoder as an image feature code for subsequent video processing, thereby improving the transmission efficiency to the decoder and reducing the transmission delay and bandwidth waste.

[0051] In some embodiments, the reserved values ​​and the one or more edge values ​​of the feature matrix for each channel are compressed in a pre-defined manner, and the number of rows to be compressed and the number of columns to be compressed as well as the number of compressed reserved values ​​and the one or more edge values ​​of the feature matrix for each channel are determined. Encoding Compression is Pre-configured This is done according to a predefined method, such as linear quantization based on bit depth values. 。 The preset method is not limited to linear quantization, and may be selected according to actual needs, such as vector quantization, and is not limited to the above example.

[0052] In some embodiments, for each channel feature matrix, Size index is determined according to the reserved value of the feature matrix of each channel and the one or more edge values ​​and the preset bit depth value, and the reserved value of the feature matrix of each channel and the compressed one or more edge values ​​correspond to the feature matrix of each channel. Size index Depending on quantum The feature matrix of each channel is compressed into a compressed reserved value and one or more compressed edge values, and the quantized reserved value and one or more quantized edge values ​​are is obtained For example, if the reserved values ​​of the channel feature matrix and the edge values ​​of 1 or more have a maximum value A and a minimum value B, the quantization level is M=2. BD Here, BD( Pre-configured The bit depth) value is a base 2 exponent, Size index teeth

number

[0053] The method of the above embodiment can achieve further data compression, improve the compression ratio, further improve the efficiency of transmission to the decoder, and reduce the waste of bandwidth.

[0054] In some embodiments, the decoder processes the received data Decoding The number of one or more rows to be compressed and the number of rows to be compressed 1 or more In addition to the number of columns, the decoder obtains the compressed reservation values ​​and one or more compressed edge values ​​of the feature matrix for each channel. The decoder obtains the number of rows to be compressed, the number of columns to be compressed, and According to the above, we pad the reserved values ​​of the feature matrix of each channel with edge values ​​of 1 or more. Recover the feature matrix for each channel. Should The number of rows (one or more) comprises a number of top rows (one or more) to compress and a number of bottom rows (one or more) to compress. The number of columns (one or more) to compress is made up of a number of left columns (one or more) to compress and a number of right columns (one or more) to compress.

[0055] For example, for channel i, the value

number

number

number

number

number

number

[0056] Furthermore, in some embodiments, as shown in FIG. 3, the processing method after the decoder receives the data comprises steps S302 to S306.

[0057] In step S302, the decoder Decoding The number of one or more rows to be compressed and the number of rows to be compressed 1 or more In addition to the number of columns, we obtain the compressed reserve value of the feature matrix for each channel and one or more compressed edge values.

[0058] In step S304, the decoder extracts the decompressed reserved values ​​of the feature matrix for each channel and Unpacked 1 or more NoE To obtain the edge value, the compressed reserved values ​​of the feature matrix for each channel and Compressed 1 or more Above Decompress the edge values.

[0059] When compressing the image using the method described in the above embodiment, the decoder uses, for example, an inverse quantization method. Unzip The data received by the decoder also includes the quantization parameters ( Size index For example, for the feature matrix of each channel, Size index are the compressed reserved values ​​of the feature matrix for each channel, and Compressed 1 or more NoE The decompressed reserved values ​​of the feature matrix for each channel and Unpacked 1 or more NoE Edge value but acquisition Be do.

[0060] In step S306, for the feature matrix of each channel, the decoder performs a compression around the decompressed reserved value of the feature matrix of each channel according to the number of one or more rows to be compressed and the number of one or more columns to be compressed. To The corresponding decompressed reserved value Solution Frozen 1 or more Edge Value Padding Then, we obtain the feature matrix for each channel.

[0061] In some embodiments, for the feature matrix of each channel, the decoder performs a compression around the compressed reserve values ​​of the feature matrix of each channel based on a number of one or more rows to compress and a number of one or more columns to compress to obtain a compressed feature matrix of each channel. To With compressed edge values ​​greater than or equal to 1 Padding Then, the compressed feature matrix is ​​decompressed to obtain the feature matrix for each channel.

[0062] In the method of the above embodiment, the compression by quantization is decoder of Decoding This is advantageous in improving the efficiency and subsequent processing of the decompressed data.

[0063] An application example will be described below.

[0064] In this application example, Cascade RCNN ResNet101 uses an RGB image of size 5472 x 3078 x 3 as input, encoding The process is as follows:

[0065] 1. Extract 64x384x672 hidden layer features after the first convolution normalization, and the feature matrix of the first channel

number

[0066] 2. Feature matrix for each channel

number

number

[0067] 3. Reserved values ​​of feature matrices for each channel under the condition of BD=8

number

number

number

number

[0068] The decoding process is as follows.

[0069] 1. Bitstream S of Decoding Then, quantize was done Reserved Value

number

number

[0070] 2. bottom and the quantized edge value on the right

number

number

number

[0071] 3. Feature Matrix

number

[0072] Before the edge addition operation, quantization was done Reserved Value

number

number

[0073] The image processing method of the present disclosure can be implemented on an encoder. The structure of an encoder according to an embodiment of the present invention will now be described with reference to FIG.

[0074] 4 is a structural diagram of an encoder according to an embodiment of the present invention. As shown in FIG. 4, the encoder 40 of the present embodiment includes a module 410 for determining information to be compressed, a module 420 for determining reserved values, a module 430 for compressing edge values, and a module 440 for encoding.

[0075] The information to be compressed determination module 410 performs the following for each channel feature matrix of the image output from the intermediate layer of the neural network: At the edge of the feature matrix One or more rows with the same feature value and one or more columns with the same feature values ​​as Let, be one or more rows to be compressed and one or more columns to be compressed. Decide where the intermediate layer is a layer between the input layer and the output layer.

[0076] In some embodiments, the information to be compressed determination module 410 selects a feature matrix of a channel of an image output from an intermediate layer of a neural network as a feature matrix to be detected, detects one or more rows having the same feature value from the first row of the feature matrix to be detected downward and from the last row of the feature matrix to be detected upward as one or more rows to be compressed of the feature matrix to be detected, detects one or more columns having the same feature value from the first column of the feature matrix to be detected backward and from the last column of the feature matrix to be detected forward as one or more columns to be compressed of the feature matrix to be detected, and sets one or more rows of the feature matrix of each channel that are in the same position as the one or more rows to be compressed of the feature matrix to be detected as one or more rows to be compressed of the feature matrix of each channel. Decide and determining to detect, as one or more columns to be compressed of the feature matrix of each channel, one or more columns of the feature matrix of each channel that are in the same position as one or more columns to be compressed of the feature matrix to be detected.

[0077] In some embodiments, the information to compress determination module 410 is configured as follows. (1) For a feature matrix to be detected, initialize the number of top rows to be compressed to 0; (2) determine whether the feature values ​​of the first row are the same, and if so, increase the number of top rows to be compressed by one and continue with the next row as the current row; (3) determine whether the feature values ​​of the current row are the same as the previous row, and if so, increase the number of top rows to be compressed by one and continue with the next row as the current row; (4) Repeat step (3) until the feature value of the current row is different from the feature value of the previous row, to obtain one or more top rows to compress and the number of one or more top rows to compress. , Inspection The feature matrix to be output is Vertical The matrix obtained by mirroring in the direction is the new feature matrix to be detected. Steps (1) from (4) to get one or more top rows to compress and the number of one or more top rows to compress, which are then used as one or more bottom rows to compress and the number of one or more bottom rows to compress.

[0078] In some embodiments, the information to compress determination module 410 includes , Inspection The matrix obtained by transposing the feature matrix to be output is As a new feature matrix to be detected, Perform steps (1) to (4) death , take one or more top rows to compress and the number of one or more top rows to compress, which are then used as one or more left columns to compress and the number of one or more left columns to compress. Be , the feature matrix to be detected is transposed and the transposed matrix is Vertical By mirroring in the direction acquisition The matrix is the new feature matrix to be detected. Execute steps (1) to (4) to find one or more top Rows and one or more to be compressed top Get the number of rows and It is configured to , which are then used as the number of one or more right columns to compress and the number of one or more right columns to compress.

[0079] The reserved value determination module 420 is configured to remove features of one or more rows to be compressed and one or more columns to be compressed in the feature matrix of each channel, and reserve the remaining feature values ​​as reserved values ​​of the feature matrix of each channel.

[0080] The edge value compression module 430 is configured to compress, for the feature matrix of each channel, feature values ​​of one or more rows to be compressed and one or more columns to be compressed to obtain one or more edge values ​​of the feature matrix of each channel.

[0081] In some embodiments, the edge value compression module 430 is configured to represent feature values ​​of one or more rows to be compressed and one or more columns to be compressed as one or more edge values ​​of the feature matrix for each channel.

[0082] Encoding The module 440 determines the number of rows to be compressed and the number of columns to be compressed, as well as the reserved values ​​and the one or more edge values ​​of the feature matrix for each channel. Encoding Then, the decoder decodes the feature matrix for each channel. Encoding The device is configured to send the results of

[0083] In some embodiments, Encoding is the entropy Encoding It is.

[0084] In some embodiments, Encoding The module 440 includes channel The reserved values ​​and one or more edge values ​​of the feature matrix of are compressed in a predetermined manner, and each channel The compressed reserved values ​​of the feature matrix and Compressed 1 or more NoE The number of rows to be compressed and the number of columns to be compressed are Encoding The device is configured to:

[0085] In some embodiments, Encoding The module 440 includes channel In the feature matrix of channel The reserved values ​​of the feature matrix and 1 or more edge values ​​and Pre-configured According to the bit depth value, channel corresponds to the feature matrix Size index Determine each channel The reserved values ​​and one or more edge values ​​of the feature matrix channel corresponds to the feature matrix Size index quantize the quantized reserve value and the quantization 1 or more The edge value channel The compressed reserved values ​​and compressed 1 or more The edge value is obtained as the edge value.

[0086] The encoder and decoder of this embodiment of the present disclosure can be implemented by various computing devices or computer systems, which are described below with reference to Figures 5 and 6. For the structure of the decoder, reference can be made to the structure of the encoder, which will not be repeated here.

[0087] 5 is a structural diagram of an encoder according to an embodiment of the present invention. Referring to FIG. 5, the encoder 50 of this embodiment includes a memory 510 and a processor 520 connected to the memory 510, the processor 520 being configured to execute the image processing method according to any of the embodiments of the present disclosure based on instructions stored in the memory 510.

[0088] Here, the memory 510 may comprise, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory stores an operating system, applications, a boot loader, databases and other programs.

[0089] 6 is a structural diagram of an encoder according to another embodiment of the present invention. As shown in FIG. 6, an encoder 60 of this embodiment includes a memory 610 and a processor 620 similar to the memory 510 and the processor 520, respectively. Encoder 60may further include an input / output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650, the memory 610, and the processor 620 may be connected, for example, via a bus 660. Here, the input / output interface 630 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, a touch screen, etc. The network interface 640 provides a connection interface to various network devices, and can be connected, for example, to a database server or a cloud storage server. The storage interface 650 provides a connection interface to an external storage device such as an SD card and a USB flash disk.

[0090] The present invention also provides an image processing system, which is described below with reference to FIG.

[0091] 7 is a structural diagram of an image processing system according to some embodiments of the present disclosure. As shown in FIG. 7, the system 7 of the present embodiment includes an encoder 40 / 50 / 60 according to any of the above-mentioned embodiments and a decoder 72.

[0092] The decoder 72 converts the received data into Decoding And compress it. 1 Number of rows or more and number of columns or more to be compressed ,and Obtain compressed reserve values ​​and one or more compressed edge values ​​of the feature matrix for each channel. death , decompressing the compressed reserved values ​​and one or more compressed edge values ​​of the feature matrix of each channel, Unzip and for each channel, the number of rows to be compressed and Should be compressed According to the number of columns, one or more, around the unzipped reserved value of the feature matrix of each channel, one or more unzipped edge values ​​corresponding to the unzipped reserved value are Padding to obtain the feature matrix for each channel. It is configured as follows: .

[0093] In some embodiments, the received data corresponds to a feature matrix for each channel. Size index further comprising: a reserved value and one or more edge values ​​of the feature matrix for each channel; Pre-configured For each channel feature matrix, the decoder 72 determines a bit depth value corresponding to the feature matrix of each channel. Size index Multiply the compressed reserved value of the feature matrix of each channel and one or more compressed edge values ​​by Unpacked 1 or more NoE The method is configured to obtain a threshold value.

[0094] In some embodiments, the encoder is located on an edge computing node and the decoder is located on the cloud.

[0095] The present disclosure further provides a method for storing a computer program. Non-transient A computer readable storage medium is provided, which when executed by a processor implements the image processing method of any one of the preceding embodiments.

[0096] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as a method, a system, or a computer program product. Thus, the embodiments of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment containing both hardware and software elements. Furthermore, the present disclosure may be implemented by one or more processors having computer usable program code embodied therein. Non-transient It may take the form of a computer program product embodied on a storage medium (including but not limited to disk storage devices, CD-ROMs, optical storage devices, etc.).

[0097] The present disclosure will be described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each process and / or block in the flowcharts and / or block diagrams, and combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. The computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or a processor of other programmable data processing apparatus such that the instructions, executed by a processor of the computer or other programmable data processing apparatus, generate means for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams to generate a machine.

[0098] Computer program instructions may be stored in a computer readable storage device to cause one or more of the flowcharts to be executed. flow and / or may be stored in a computer-readable storage device capable of instructing a computer or other programmable data processing apparatus to operate in a particular manner to produce an article of manufacture having instructions that implement the functions specified in one or more blocks of the block diagram.

[0099] These computer program instructions can be loaded into a computer or other programmable device, so that the instructions executed on the computer or other programmable device provide steps for implementing functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams, and can execute a series of operation steps on the computer or other programmable device to generate a computer-implemented process. The above description is merely a preferred embodiment of the present disclosure, and is not intended to limit the present disclosure. Any modifications, substitutions, improvements, etc. within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method for a feature matrix of each channel of an image output from an intermediate layer of a neural network, the image processing method comprising: determining one or more rows having the same feature value and one or more columns having the same feature value at an edge of the feature matrix as one or more rows to be compressed and one or more columns to be compressed, where the intermediate layer is a layer between an input layer and an output layer; determining one or more rows having the same feature value and one or more columns having the same feature value; removing the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed and reserving remaining feature values ​​as reserved values ​​of the feature matrix for each of the channels; compressing the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed to obtain one or more edge values ​​of the feature matrix for each of the channels; encoding the reserved values ​​and the one or more edge values ​​of the feature matrix for each channel, as well as the number of rows to be compressed and the number of columns to be compressed, sending a result of the encoding to a decoder, and restoring the feature matrix for each channel by the decoder.

2. Determining one or more rows having the same feature value at an edge of the feature matrix, and one or more columns having the same feature value, Selecting a feature matrix of a channel of the image output from the intermediate layer of the neural network as a feature matrix to be detected; detecting one or more rows having the same feature value as the one or more rows to be compressed of the feature matrix to be detected from a first row downward and from a last row upward, respectively, of the feature matrix to be detected, and detecting one or more columns having the same feature value as the one or more columns to be compressed of the feature matrix to be detected from a first column backward and from a last column forward of the feature matrix to be detected; 2. The image processing method according to claim 1 , further comprising: determining one or more rows of the feature matrix of each channel that are in the same position as the one or more rows to be compressed of the feature matrix to be detected as one or more rows of the feature matrix of each channel to be compressed; and determining one or more columns of the feature matrix of each channel that are in the same position as the one or more columns to be compressed of the feature matrix to be detected as one or more columns of the feature matrix of each channel to be compressed.

3. Encoding the reserved values ​​and the one or more edge values ​​of the feature matrix for each channel, and the number of the one or more rows to be compressed and the number of the one or more columns to be compressed, compressing the reserved values ​​and the one or more edge values ​​of the feature matrix for each of the channels in a predetermined manner; 2. The image processing method of claim 1, further comprising: encoding the compressed reservation values ​​and the compressed one or more edge values ​​of the feature matrix for each channel, as well as the number of the one or more rows to be compressed and the number of the one or more columns to be compressed.

4. Compressing the reserved values ​​and the one or more edge values ​​of the feature matrix for each of the channels in the predetermined manner includes: determining, for the feature matrix of each of the channels, a magnitude indicator corresponding to the feature matrix of each of the channels according to the reservation value of the feature matrix of each of the channels, the one or more edge values, and a preset bit depth value; quantizing the reserved values ​​and the one or more edge values ​​of the feature matrix for each channel according to the magnitude indicator corresponding to the feature matrix for each channel to obtain quantized reserved values ​​and quantized one or more edge values ​​as the compressed reserved values ​​and the compressed one or more edge values ​​of the feature matrix for each channel.

5. Detecting one or more rows having the same feature value from a first row of the feature matrix to be detected downward and from a last row of the feature matrix to be detected upward, respectively, comprises: (1) initializing to zero a number of one or more top rows to be compressed for the feature matrix to be detected; (2) determining whether the feature values ​​of the first row are the same, and if so, incrementing the number of the one or more top rows to be compressed by one and continuing with the next row as the current row; (3) determining whether the feature value of the current row is the same as the previous row, and if so, increasing the number of the one or more top rows to be compressed by one and continuing with the next row as the current row; (4) repeating step (3) until the feature value of the current row is different from the feature value of the previous row to obtain one or more top rows to compress and the number of the top rows to compress; 3. The image processing method of claim 2, further comprising: performing steps (1) to (4) on a matrix obtained by vertically mirroring the feature matrix to be detected as a new feature matrix to be detected, to obtain one or more top rows to be compressed and the number of the one or more top rows to be compressed, which are used as one or more bottom rows to be compressed and the number of the one or more bottom rows to be compressed.

6. Detecting one or more columns having the same feature value from a first column of the feature matrix to be detected backward and from a last column of the feature matrix to be detected forward, respectively, comprises: Perform steps (1) to (4) on a matrix obtained by transposing the feature matrix to be detected as a new feature matrix to be detected to obtain one or more top rows to be compressed and the number of the one or more top rows to be compressed, which are used as one or more left columns to be compressed and the number of the one or more left columns to be compressed; and performing steps (1) to (4) to obtain a matrix obtained by vertically mirroring a transposed matrix of the feature matrix to be detected as a new feature matrix to be detected, thereby obtaining one or more top rows to be compressed and the number of the one or more top rows to be compressed, which are used as one or more right columns to be compressed and the number of the one or more right columns to be compressed.

7. The image processing method of claim 1, wherein compressing the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed to obtain one or more edge values ​​of the feature matrix of each channel includes representing the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed with a single value as the one or more edge values ​​of the feature matrix of each channel.

8. An image processing method according to claim 1, wherein the encoding is entropy encoding.

9. decoding, by the decoder, the received data to obtain the one or more numbers of rows to be compressed and the one or more numbers of columns to be compressed as well as the compressed reserve values ​​and the compressed one or more edge values ​​of the feature matrix for each of the channels; decompressing, by the decoder, the compressed reserve values ​​of the feature matrix and the one or more compressed edge values ​​for each of the channels to obtain decompressed reserve values ​​of the feature matrix and the one or more decompressed edge values ​​for each of the channels; 4. The image processing method of claim 3, further comprising: for the feature matrix for each channel, padding, by the decoder, the one or more decompressed edge values ​​corresponding to the decompressed reserved values ​​around the decompressed reserved values ​​of the feature matrix for each channel in accordance with a number of the one or more rows to be compressed and a number of the one or more columns to be compressed to obtain the feature matrix for each channel.

10. The received data further includes a magnitude index corresponding to the feature matrix of each of the channels, the magnitude index being determined according to the reserved value of the feature matrix of each of the channels, the one or more edge values, and a preset bit depth value; Decompressing, by the decoder, the compressed reserve values ​​and the compressed one or more edge values ​​of the feature matrix for each of the channels, 10. The image processing method of claim 9, comprising, for the feature matrix for each channel, multiplying by the decoder the magnitude indicator corresponding to the feature matrix for each channel by the compressed reserve value and the one or more compressed edge values ​​of the feature matrix for each channel, respectively, to obtain the decompressed reserve value and the one or more decompressed edge values ​​of the feature matrix for each channel.

11. an information to be compressed determination module for determining, for a feature matrix of each channel of an image output from an intermediate layer of a neural network, one or more rows having the same feature value at an edge of the feature matrix and one or more columns having the same feature value as one or more rows to be compressed and one or more columns to be compressed, the intermediate layer being a layer between an input layer and an output layer; a reserved value determination module for deleting the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed from the feature matrix of each channel, and reserving the remaining feature values ​​as reserved values ​​of the feature matrix of each channel; an edge value compression module for compressing the feature values ​​of the one or more rows to be compressed and the one or more columns to be compressed to obtain one or more edge values ​​of the feature matrix for each of the channels; an encoding module that encodes the reserved values ​​and the one or more edge values ​​of the feature matrix for each channel, as well as the one or more numbers of rows to be compressed and the one or more numbers of columns to be compressed, sends a result of the encoding to a decoder, and restores the feature matrix for each channel by the decoder.

12. A processor; and a memory connected to the processor for storing instructions which, when executed by the processor, cause the processor to perform the image processing method of any one of claims 1 to 8.

13. 11. A non-transitory computer readable storage medium comprising a computer program stored thereon, the program causing, when executed by a processor, to perform the steps of the image processing method according to any one of claims 1 to 10.

14. An image processing system comprising the encoder according to claim 11 or 12 and a decoder, The decoder reconstructs the feature matrix for each of the channels in response to received data transmitted by the encoder.

15. The image processing system of claim 14, wherein the decoder decodes the received data to obtain the compressed reserved values ​​and the one or more compressed edge values ​​of the feature matrix for each channel in addition to the number of the one or more rows to be compressed and the number of the one or more columns to be compressed, decompresses the compressed reserved values ​​and the one or more compressed edge values ​​of the feature matrix for each channel, obtains the decompressed reserved values ​​and the one or more decompressed edge values ​​of the feature matrix for each channel, and pads the one or more decompressed edge values ​​corresponding to the decompressed reserved values ​​around the decompressed reserved values ​​of the feature matrix for each channel in accordance with the number of the one or more rows to be compressed and the number of the one or more columns to be compressed to obtain the feature matrix for each channel.

16. The received data further includes a magnitude index corresponding to the feature matrix of each of the channels, the magnitude index being determined according to the reserved value of the feature matrix of each of the channels, the one or more edge values, and a preset bit depth value; 16. The image processing system of claim 15, wherein the decoder multiplies, for the feature matrix for each channel, the magnitude indicator corresponding to the feature matrix for each channel by the compressed reserve value and the one or more compressed edge values ​​of the feature matrix for each channel, respectively, to obtain the decompressed reserve value and the one or more decompressed edge values ​​of the feature matrix for each channel.

Images