Image processing method, device and storage medium
By combining feature concatenation and deconvolution operations with transformation and encoding processing, the problems of high computational cost and low accuracy in multi-scale feature compression technology are solved, achieving efficient image feature compression and decompression, which is suitable for low-end devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CORP LTD
- Filing Date
- 2022-09-30
- Publication Date
- 2026-07-24
AI Technical Summary
Existing multi-scale feature compression techniques are computationally intensive, have a large number of parameters, consume a lot of memory, have low accuracy and low compression rate, making them difficult to apply effectively on low-end devices.
Equivalent image feature information is generated by feature stitching, the equivalent number of channels is determined, and multiple convolution processes are performed using the feature compression module. Image compressed data is generated by combining transformation and encoding. The feature decompression module performs reverse processing to obtain equivalent image feature information. Convolution operation is canceled and encoder is used to reduce computational complexity. GAP operation is canceled to reduce information loss.
It reduces computational complexity, improves processing accuracy, enhances compression ratio, reduces memory usage, and improves user experience.
Smart Images
Figure CN115512205B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, apparatus, and storage medium. Background Technology
[0002] Multi-scale feature compression technology is widely used in various application scenarios such as target recognition and target segmentation. For example, after acquiring images or videos, IoT (Internet of Things) devices use extraction networks to obtain features representing the images or videos. After quantizing and entropy encoding the features, they are sent to the server for entropy decoding and dequantization to recover the features. The recovered features can then be used to run other artificial intelligence tasks, such as restoring images or performing object detection. However, existing multi-scale feature compression and decompression technologies suffer from drawbacks such as high computational cost, numerous parameters, large memory consumption, low accuracy, and low compression ratio. Summary of the Invention
[0003] In view of this, one technical problem to be solved by the present invention is to provide an image processing method, apparatus and storage medium.
[0004] According to a first aspect of this disclosure, an image processing method is provided, comprising: concatenating multiple image feature information to generate equivalent image feature information; determining the equivalent number of channels corresponding to the equivalent image feature information, and determining the number of main convolution channels corresponding to a feature compression module based on the equivalent number of channels; performing multiple convolution processes on the equivalent image feature information using the feature compression module and based on the number of main convolution channels and a preset number of convolution channels to generate dimensionality-reduced image feature information corresponding to the equivalent image feature information; performing transformation and encoding processing on the dimensionality-reduced image feature information using the feature compression module to generate compressed image data; performing reverse processing on the compressed image data using a feature decompression module corresponding to the processing performed by the feature compression module to obtain the equivalent image feature information; and performing segmentation processing on the equivalent image feature information to obtain the multiple image feature information.
[0005] Optionally, the dimensions of the image feature information include: number of channels, image width, and image height; determining the equivalent number of channels corresponding to the equivalent image feature information includes: obtaining the minimum image width and minimum height among the dimensions of the plurality of image feature information, as the equivalent image width and equivalent image height among the dimensions of the equivalent image feature information; calculating the product of the equivalent image width and the equivalent image height; and taking the quotient of the total data volume of the equivalent image feature information and the product as the equivalent number of channels.
[0006] Optionally, determining the number of main convolutional channels corresponding to the feature compression module based on the equivalent number of channels includes setting the number of main convolutional channels to the equivalent number of channels.
[0007] Optionally, the feature compression module includes multiple first convolutional layers and at least one second convolutional layer; the step of using the feature compression module and performing multiple convolutional processes on the equivalent image feature information based on the number of main convolutional channels and a preset number of convolutional channels to generate dimensionality-reduced image feature information corresponding to the equivalent image feature information includes: setting the number of input channels and the number of output channels of the first convolutional layer to the number of main convolutional channels; setting the number of input channels of the second convolutional layer to the number of main convolutional channels and the number of output channels to the preset number of convolutional channels; wherein, the preset number of convolutional channels includes: 8, 16, 32; and sequentially using the first convolutional layer and the second convolutional layer to perform multiple convolutional processes on the equivalent image feature information to generate the dimensionality-reduced image feature information.
[0008] Optionally, the feature compression module includes: a first transformation processing layer and a first encoding layer; the step of using the feature compression module to perform transformation processing and encoding processing on the dimensionality-reduced image feature information to generate image compressed data includes: using the first transformation processing layer to perform Discrete Cosine Transform (DCT) processing on the dimensionality-reduced image feature information, and using the first encoding layer to perform compression encoding processing on the dimensionality-reduced image feature information after DCT processing to generate the image compressed data.
[0009] Optionally, the reverse processing performed on the image compressed data using the feature decompression module, corresponding to the processing performed by the feature compression module, includes: using the feature decompression module to perform decoding and inverse transformation processing on the image compressed data to generate dimensionality-reduced image feature information; using the feature decompression module and based on the number of main convolution channels and a preset number of convolution channels to perform multiple deconvolution processes on the dimensionality-reduced image feature information to generate equivalent image feature information corresponding to the dimensionality-reduced image feature information.
[0010] Optionally, the feature decompression module includes: a second encoding layer and a second transform processing layer; the step of using the feature decompression module to perform inverse encoding and inverse transform processing on the image compressed data to generate dimensionality-reduced image feature information includes: using the second encoding layer to perform inverse compression encoding processing on the image compressed data, and using the second transform processing layer to perform inverse DCT processing on the image compressed data after inverse compression encoding processing to generate the dimensionality-reduced image feature information.
[0011] Optionally, the step of using the feature decompression module and performing multiple deconvolution processes on the dimensionality-reduced image feature information based on the number of main convolution channels and a preset number of convolution channels includes: setting the number of input channels of the third convolutional layer to the preset number of convolution channels and the number of output channels to the number of main convolution channels; setting the number of input channels and the number of output channels of the fourth convolutional layer to the number of main convolution channels; and sequentially using the third convolutional layer and the fourth convolutional layer to perform multiple deconvolution processes on the dimensionality-reduced image feature information to generate equivalent image feature information.
[0012] Optionally, a feature extraction module is used to perform feature extraction processing on the image to obtain multiple image feature information corresponding to the image;
[0013] Optionally, the feature extraction module includes a Feature Pyramid Network (FPN) model; the step of using the feature extraction module to perform feature extraction processing on the image and obtain multiple image feature information corresponding to the image includes: processing the image input using the FPN model to generate the multiple image feature information; wherein the image width dimension and image height dimension of the multiple image feature information are both smaller than the corresponding image width dimension and image height dimension in the image, respectively; and the image width dimension and image height dimension of the multiple image feature information decrease sequentially.
[0014] According to a second aspect of this disclosure, an image processing apparatus is provided, comprising: a feature stitching module for stitching multiple image feature information to generate equivalent image feature information; a channel number determination module for determining the equivalent channel number corresponding to the equivalent image feature information, and determining the main convolution channel number corresponding to a feature compression module based on the equivalent channel number; a feature compression module for performing multiple convolution processes on the equivalent image feature information based on the main convolution channel number and a preset convolution channel number to generate dimensionality-reduced image feature information corresponding to the equivalent image feature information; performing transformation and encoding processing on the dimensionality-reduced image feature information using the feature compression module to generate compressed image data; a feature decompression module for performing reverse processing on the compressed image data corresponding to the processing performed by the feature compression module to obtain the equivalent image feature information; and a feature segmentation module for segmenting the equivalent image feature information to obtain the multiple image feature information.
[0015] Optionally, the dimensions of the image feature information include: number of channels, image width, and image height; the channel number determination module is used to obtain the minimum image width and minimum height among the dimensions of the multiple image feature information, as the equivalent image width and equivalent image height among the dimensions of the equivalent image feature information; calculate the product of the equivalent image width and the equivalent image height; and take the quotient of the total data volume of the equivalent image feature information and the product as the equivalent number of channels.
[0016] Optionally, the channel number determination module is used to set the main convolution channel number as the equivalent channel number.
[0017] Optionally, the feature compression module includes multiple first convolutional layers and at least one second convolutional layer; the feature compression module is used to set the number of input channels and output channels of the first convolutional layer to the number of main convolutional channels, and set the stride of the first convolutional layer to a value greater than 1; set the number of input channels of the second convolutional layer to the number of main convolutional channels, the number of output channels to the preset number of convolutional channels, and set the stride of the first convolutional layer to 1; wherein, the preset number of convolutional channels includes: 8, 16, 32; the first convolutional layer and the second convolutional layer are used sequentially to perform multiple convolutional processes on the equivalent image feature information to generate the dimensionality-reduced image feature information.
[0018] Optionally, the feature compression module includes: a first transformation processing layer and a first encoding layer; the feature conversion module is used to perform Discrete Cosine Transform (DCT) processing on the dimensionality-reduced image feature information using the first transformation processing layer, and to perform compression encoding processing on the dimensionality-reduced image feature information after DCT processing using the first encoding layer to generate the image compressed data.
[0019] Optionally, the feature decompression module is used to perform inverse encoding and inverse transformation processing on the image compressed data to generate dimensionality-reduced image feature information; and to perform multiple deconvolution processes on the dimensionality-reduced image feature information based on the number of main convolution channels and a preset number of convolution channels to generate equivalent image feature information corresponding to the dimensionality-reduced image feature information.
[0020] Optionally, the feature decompression module includes: a second encoding layer and a second transform processing layer; the feature decompression module is used to perform inverse compression encoding processing on the image compressed data using the second encoding layer, and to perform inverse DCT processing on the image compressed data after inverse compression encoding processing using the second transform processing layer, to generate the dimensionality-reduced image feature information.
[0021] Optionally, the feature decompression module is used to set the number of input channels of the third convolutional layer to the preset number of convolutional channels and the number of output channels to the number of main convolutional channels, and to set the stride of the third convolutional layer to 1; set the number of input channels and the number of output channels of the fourth convolutional layer to the number of main convolutional channels, and to set the stride of the fourth convolutional layer to a value greater than 1; and sequentially use the third convolutional layer and the fourth convolutional layer to perform multiple deconvolution processes on the dimensionality-reduced image feature information to generate equivalent image feature information.
[0022] Optionally, the feature extraction module is used to perform feature extraction processing on the image to obtain multiple image feature information corresponding to the image;
[0023] Optionally, the feature extraction module includes a Feature Pyramid Network (FPN) model; the feature extraction module is used to process the image input using the FPN model to generate the plurality of image feature information; wherein the image width dimension and image height dimension of the plurality of image feature information are both smaller than the corresponding image width dimension and image height dimension in the image, respectively; and the image width dimension and image height dimension of the plurality of image feature information decrease sequentially.
[0024] According to a third aspect of this disclosure, an image processing apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method described above based on instructions stored in the memory.
[0025] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions which are executed by a processor using the method described above.
[0026] The image processing method, apparatus, and storage medium disclosed herein reduce computational complexity by eliminating convolution operations in feature stitching and feature segmentation reconstruction; reduce lost information by eliminating GAP operations and improve processing accuracy by preventing multi-level features from being recovered through other features; improve compression ratio by using convolution processing to reduce data volume and further compress channels; effectively reduce memory usage by eliminating FC operations and reducing convolution processing; and improve user experience. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0028] Figure 1 This is a schematic diagram of the application architecture of multi-scale feature processing in existing technologies.
[0029] Figure 2 This is a schematic flowchart of an embodiment of the image processing method according to the present disclosure;
[0030] Figure 3 This is a schematic diagram of the application architecture of an embodiment of the image processing method according to the present disclosure;
[0031] Figure 4 This is a schematic diagram of a module of an image processing apparatus according to an embodiment of the present disclosure;
[0032] Figure 5 This is a schematic diagram of a module according to another embodiment of the image processing apparatus of the present disclosure;
[0033] Figure 6 This is a schematic diagram of a module according to yet another embodiment of the image processing apparatus of the present disclosure. Detailed Implementation
[0034] The present disclosure will now be described more fully with reference to the accompanying drawings, which illustrate exemplary embodiments of the present disclosure. The technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative effort are within the scope of protection of the present disclosure.
[0035] Among the related technologies known to the inventor, the amount of multi-scale feature data extracted using multi-scale feature technology is significantly increased compared to the amount of original image data. Multi-scale feature compression methods have disadvantages such as large computational load, large number of parameters, large memory consumption, low accuracy, and low compression rate.
[0036] Taking the FPN structure in deep learning as an example, and object recognition as an example, assuming the original image size is H 640 and W 800, the data volume of the original image is 640*800*3; the multi-scale features provided by FPN are P2, P3, P4, and P5, and the total data volume of its multi-scale features is 256*(20*25+40*50+80*100+160*200); the total data volume of the multi-scale features is 11.8 times the data volume of the original image; when the cloud system needs to collaborate, and the edge wants to transmit the already calculated multi-scale features for cloud use, the transmitted features are larger than the data volume of the original image, so the features must be compressed more.
[0037] like Figure 1As shown, the feature extraction network extracts multi-level features P2…P5, which have different sizes. Since P2…P5 have different sizes, the feature rearrangement module first adjusts the size of each feature to make them the same. The feature fusion module consists of an SE (Squeeze-and-Excitation) layer, which fuses multiple features into one feature. The feature compression module further compresses the size of the fused feature and then performs tool compression. The feature decompression module recovers information from the bitstream and then recovers the fused feature through the network. The recovered feature directly represents layer P5. Features P4 are recovered through convolution with a 2x upsampling of features P5, and features P3 are recovered through convolution with a 4x upsampling of features P5 and a 2x upsampling of features P4. Features P2 are recovered using a similar method.
[0038] The image processing techniques described above that employ multi-scale feature extraction have the following problems:
[0039] 1. The computational load is very high. For example, in the process of feature rearrangement, the feature size needs to be adjusted to the same size after each layer performs a 1x1 convolution on the features; the feature recovery process requires multiple convolutions and multiple upsampling operations; feature fusion requires GAP (Global Average Pooling) and two FC (Fully Connected Layers); feature compression and decompression also require the use of convolution functions, employing highly complex algorithms.
[0040] 2. Low accuracy. Because multi-level features reflect the characteristics of images at different scales, the fused features actually represent more directly the P5 features, while other features are recovered from the P5 features according to the rules.
[0041] 3. Another reason for the low accuracy is the GAP processing. Since GAP takes the maximum value globally, it is equivalent to taking the most significant feature among the four scale features as the fusion feature. Therefore, the fusion feature does not completely match the original P5 feature. When superimposing other features, it is necessary to recover from P5. Therefore, the information lost during the compression process is extremely difficult to recover.
[0042] 4. Accuracy and compression ratio are closely related. The compression ratio needs to be considered while meeting a certain level of accuracy. When the accuracy is not high, a lower compression ratio should be set to match the accuracy requirements.
[0043] 5. The model has large parameters and is not suitable for low-end devices. This is because the decompression end is usually located in the cloud, while the compression end typically has small computing power and limited memory. The SE operation consists of GAP + squeeze (FC operation, channel dimensionality reduction) + excitation (image processing method operation, channel dimensionality increase). The number of FC parameters is very large, and the number of parameters of the many 1x1 convolutions that are stacked, aligned, and rearranged is also very large. Therefore, the actual memory requirements for the compression end are very high.
[0044] Figure 2 This is a schematic flowchart of an embodiment of the image processing method according to the present disclosure, as shown below. Figure 2 As shown:
[0045] Step 201: Multiple image feature information is spliced together to generate equivalent image feature information.
[0046] In one embodiment, multiple methods can be used to obtain multiple image feature information. For example, a feature extraction module can be used to perform feature extraction processing on the image to obtain multiple image feature information corresponding to the image. The image disclosed herein can be a picture, a photograph, a video frame from a video, etc.
[0047] The feature extraction module includes the Feature Pyramid Network (FPN) model. The FPN model processes the image input to generate multiple image feature information units. The width and height dimensions of these multiple feature units are smaller than the corresponding width and height dimensions in the original image, respectively; furthermore, the width and height dimensions decrease sequentially across the multiple feature units. These multiple feature units can be multiple vectors, which are then added together to generate equivalent image feature information.
[0048] Step 202: Determine the number of equivalent channels corresponding to the equivalent image feature information, and determine the number of main convolution channels corresponding to the feature compression module based on the number of equivalent channels.
[0049] Step 203: Use the feature compression module and perform multiple convolution processes on the equivalent image feature information based on the number of main convolution channels and the preset number of convolution channels to generate dimensionality-reduced image feature information corresponding to the equivalent image feature information.
[0050] Step 204: Use the feature compression module to transform and encode the feature information of the dimensionality-reduced image to generate compressed image data.
[0051] Step 205: The feature decompression module performs the reverse processing on the image compressed data, corresponding to the processing performed by the feature compression module, to obtain equivalent image feature information.
[0052] Step 206: Segment the equivalent image feature information to obtain multiple image feature information. Based on the lengths of the multiple representative image feature information that make up the equivalent image feature information, segment the equivalent image feature information to obtain multiple image feature information.
[0053] The feature extraction, feature fusion, feature compression, feature decompression, and feature segmentation modules of the image processing method disclosed herein are pre-trained modules and can be trained using various existing training methods.
[0054] In one embodiment, the dimensions of image feature information include the number of channels C, image width W, and image height H. The minimum image width and minimum height among the dimensions of multiple image feature information are obtained as the equivalent image width and equivalent image height in the dimensions of the equivalent image feature information. The product of the equivalent image width and equivalent image height is calculated, and the quotient of the product of the total amount of equivalent image feature information and the equivalent number of channels is taken as the equivalent number of channels. The number of main convolution channels is set as the equivalent number of channels.
[0055] like Figure 3 As shown, assuming the original image size is h640 and w800, the original image data size is 640*800*3, and the total data size of the multi-scale features is 256*(20*25+40*50+80*100+160*200), which is 11.8 times the data size of the original image.
[0056] The feature extraction module is used to extract features from the image, obtaining multiple image feature information corresponding to the image (3, h, w), namely P2, P3, P4, and P5. The multi-scale features P2, P3, P4, and P5 are not convolved or have their lengths modified; they are only concatenated to obtain the equivalent feature dimensions (equivalent image feature information) after concatenation.
[0057] Determine the minimum image width W and minimum height H among the dimensions of multiple image feature information. For example, the minimum height H is h / 32, and the minimum image width W is w / 32. C = element / (H*W), where element is the total data volume of the stitched equivalent image feature information. For example, the total data volume of the stitched equivalent image feature information is: 256*((h / 32)*(w / 32)+(h / 16)*(w / 16)+(h / 8)*(w / 8)+(h / 4)*(w / 4)), then the equivalent number of channels C = 256*85, and the dimensions of the equivalent image feature information are (256*85, h / 32, w / 32).
[0058] The CHW values of the equivalent image feature information are C = 256*85, H = h / 32, and W = w / 32, respectively. Based on the equivalent feature dimensions, the number of main convolutional channels corresponding to the feature compression module is determined, and the number of main convolutional channels is the equivalent channel C.
[0059] In one embodiment, the feature compression module includes multiple first convolutional layers and at least one second convolutional layer. The first and second convolutional layers can be various existing convolutional layers. The number of input and output channels of the first convolutional layer is set to the number of principal convolutional channels, and the stride of the first convolutional layer is set to a value greater than 1. The number of input channels of the second convolutional layer is set to the number of principal convolutional channels, and the number of output channels is set to a preset number of convolutional channels. The stride of the first convolutional layer is set to 1. The preset number of convolutional channels includes 8, 16, 32, etc. The equivalent image feature information is processed multiple times using the first and second convolutional layers sequentially to generate dimensionality-reduced image feature information.
[0060] like Figure 3 As shown, the feature compression module includes a first convolutional layer 301, 302 and a second convolutional layer 303. The number of channels in the first convolutional layers 301 and 302 is set to the equivalent number of channels C = 256 * 85, and the stride of the first convolutional layers 301 and 302 is set to 2. The number of output convolutional channels in the second convolutional layer 303 is set to a preset number of convolutional channels C', which is a very small number, such as 8, 16, 32, etc. After the first convolutional layers 301 and 302, a BN (Batch Normalization) layer and an ACT activation layer are set.
[0061] The feature compression module uses the first convolutional layer 301 and 302 with a stride of 2 to convolve the equivalent image feature information, changing the feature scale to be compressed to (C, H / 4, W / 4); the feature compression module uses the second convolutional layer 303 to perform another convolution, and the dimension of this feature to be compressed is (C', H / 4, W / 4); taking C' = 8 as an example, the actual amount of feature data to be encoded is 1 / (512*85) of the data amount of the concatenated equivalent image features.
[0062] like Figure 3 As shown, the feature compression module includes a first transform processing layer 304 and a first encoding layer 305. The first transform processing layer 304 can employ various transform processes. For example, the first transform processing layer 304 performs Discrete Cosine Transform (DCT) processing on the dimensionality-reduced image feature information, quantizes and compresses the DCT coefficients, and the first encoding layer 305 performs compression encoding processing on the DCT-processed dimensionality-reduced image feature information to generate compressed image data.
[0063] In one embodiment, performing the reverse processing corresponding to the processing performed by the feature decompression module on the image compressed data can be achieved using various methods. For example, the feature decompression module can be used to perform decoding and inverse transform processing on the image compressed data to generate dimensionality-reduced image feature information. Alternatively, the feature decompression module can be used to perform multiple deconvolution processes on the dimensionality-reduced image feature information based on the main convolution channel number and a preset convolution channel number to generate equivalent image feature information corresponding to the dimensionality-reduced image feature information.
[0064] like Figure 3 As shown, the feature decompression module includes a second encoding layer 306 and a second transform processing layer 307. The second encoding layer 306 and the second transform processing layer 307 respectively perform inverse transforms corresponding to the first encoding layer 305 and the first transform processing layer 304. For example, the second encoding layer 306 is used to perform inverse compression encoding on the image compressed data, and the second transform processing layer 307 is used to perform inverse DCT processing on the inverse compression encoded image compressed data to generate dimensionality-reduced image feature information.
[0065] In one embodiment, the feature decompression module includes at least one third convolutional layer and multiple fourth convolutional layers. A Batch Normalization (BN) layer and an Actual Transform (ACT) layer are then positioned after the third and fourth convolutional layers. The number of third and second convolutional layers is the same, and the third convolutional layer performs the inverse transform corresponding to the second convolutional layer. The number of fourth convolutional layers is the same as the first convolutional layer, and the fourth convolutional layer performs the inverse transform corresponding to the first convolutional layer.
[0066] For example, such as Figure 3 As shown, the feature decompression module includes a second convolutional layer 308 and multiple fourth convolutional layers 309 and 310. The number of channels in the third convolutional layer 308 is set to the equivalent number of channels C, and the stride of the third convolutional layer 308 is set to 1; the stride of the fourth convolutional layers 309 and 310 is set to 2. The third convolutional layer 308 and the fourth convolutional layers 309 and 310 are used sequentially to perform multiple deconvolution processes on the image features, generating equivalent image feature information. A fifth convolutional layer 311 is set to convolve the equivalent image feature information processed by ACT, ensuring that the equivalent image feature information output by the fifth convolutional layer 311 is the same as the equivalent image feature information output by the feature stitching module. The feature extraction module is used to perform feature extraction processing on the image, obtaining multiple image feature information corresponding to the image.
[0067] In one embodiment, assuming the image dimensions are h = 640 and w = 800, the image is processed by FPN in the feature extraction module, and the extracted multi-scale dimensions are: P5(256, 20, 25), P4(256, 40, 50), P3(256, 80, 10), and P2(256, 160, 200). The H of the equivalent image features after stitching is taken as the H parameter of P5 because the H parameter of P5 is the smallest. Similarly, the W of the equivalent image features is taken as the W parameter of P5 (H = 20, W = 25).
[0068] We determine C = 256*(20*25+40*50+80*100+160*200) / (20*25) = 256*85, where C is the number of channels in the equivalent image feature information. The dimension of the equivalent image feature information is CHW(256*85,20,25).
[0069] For the first convolutional layer 301, 302 is set with kernel=3, input channels 256*85, output channels 256*85, and stride=2. The first convolutional layer 301 performs a first downsampling convolution, resulting in image features with dimensions (256*85, 10, 12). The first convolutional layer 302 then performs a second downsampling convolution, resulting in image features with dimensions (256*85, 5, 6).
[0070] For the second convolutional layer 303, the kernel is set to 3, the number of input channels is 256*85, the number of output channels is 32, and the stride is 1. A third convolution is performed using the second convolutional layer 303, resulting in image features with dimensions (32, 5, 6). This feature is then subjected to a DCT transform, and the transform coefficients are quantized and compressed to obtain a bitstream. A feature decompression module is used to reverse the order of the compressed image data to obtain equivalent image feature information.
[0071] If the input image to the feature extraction module is a single-scale feature image, the feature stitching and feature segmentation modules can be omitted to reduce the amount of original features to be compressed and improve the compression ratio. For example, assuming the image dimensions are h=640, w=800, the single-scale feature extracted by the feature extraction module is t2(256, 160, 200). For the first convolutional layer 301, 302 is set with kernel=3, input channels 256, output channels 256, and stride=2. The first downsampling convolution is performed using the first convolutional layer 301, resulting in a convolutional image feature dimension of (256, 80, 100). A second downsampling convolution is performed using the first convolutional layer 302, resulting in a convolutional image feature dimension of (256, 40, 50). For the second convolutional layer 303, kernel=3, input channels 256, output channels 16, and stride=1. A third convolution is performed using the second convolutional layer 303, resulting in image features with dimensions of (16, 40, 50). This feature is then subjected to a DCT transform, and the transform coefficients are quantized and compressed to obtain a bitstream. A feature decompression module is used to reverse the order of the compressed image data to obtain equivalent image feature information.
[0072] In the image processing method of the above embodiments, multi-scale features are only stitched together to obtain the equivalent feature dimension CHW after stitching. The number of subsequent main body convolution channels C is determined according to the equivalent feature dimension. For equivalent feature compression, multi-level convolution with a stride of 2 is first used to further change the scale of the feature to be compressed to (C, H / 4, W / 4). For equivalent feature compression, another convolution is performed and the output channel is set to C'. The feature dimension to be compressed is (C', H / 4, W / 4). Taking C' = 8 and C = 256 as an example, the actual amount of feature data to be encoded is 1 / 512 of the amount of data of the equivalent feature after stitching.
[0073] For features of dimension (C', H / 4, W / 4), further feature transformations are performed, such as DCT transformation, and their coefficients are quantized and compressed. During feature decompression, a reverse operation is performed: first, a convolution is performed to restore the number of channels from C' to C, and then two deconvolutions are performed to restore the dimensions to H and W.
[0074] The image processing method in the above embodiments uses feature stitching and eliminates convolution operations, feature segmentation and reconstruction eliminates convolution operations, and the encoder replaces the VTM, significantly reducing computational complexity; the GAP operation is eliminated, and the different channels of the stitched features contain features of different scales and features of the surrounding region, which can be learned through convolution with less loss of information, and multi-level features are not recovered by other features, improving accuracy; two downsampling convolutions are used to reduce the data volume to 1 / 16 of the original data, and then further compress the channels, improving the compression ratio; the two fully connected (FC) operations of the SE operation and numerous 1x1 convolutions are eliminated, effectively reducing memory usage.
[0075] In one embodiment, such as Figure 4 As shown, this disclosure provides an image processing apparatus 40, including a feature stitching module 41, a channel number determination module 42, a feature compression module 43, a feature decompression module 44, and a feature segmentation module 45. The feature stitching module 41 stitches together multiple image feature information to generate equivalent image feature information. The channel number determination module 42 determines the equivalent number of channels corresponding to the equivalent image feature information, and determines the number of main convolution channels corresponding to the feature compression module based on the equivalent number of channels.
[0076] The feature compression module 43 performs multiple convolution processes on the equivalent image feature information based on the main convolution channel number and a preset convolution channel number, generating dimensionality-reduced image feature information corresponding to the equivalent image feature information. The feature compression module 43 then performs transformation and encoding processes on the dimensionality-reduced image feature information to generate compressed image data. The feature decompression module 44 performs the reverse processing corresponding to the processing performed by the feature compression module on the compressed image data to obtain the equivalent image feature information. The feature segmentation module 45 performs segmentation processing on the equivalent image feature information to obtain multiple image feature information.
[0077] In one embodiment, the dimensions of the image feature information include the number of channels, image width, and image height. The channel number determination module 42 obtains the minimum image width and minimum height among the dimensions of multiple image feature information, which are used as the equivalent image width and equivalent image height in the dimensions of the equivalent image feature information. The channel number determination module 42 calculates the product of the equivalent image width and the equivalent image height, and takes the quotient of the total data volume of the equivalent image feature information and the product as the equivalent number of channels. The channel number determination module 42 sets the number of main convolution channels as the equivalent number of channels.
[0078] The feature compression module 43 includes multiple first convolutional layers and at least one second convolutional layer. The feature compression module 43 sets the number of input channels and output channels of the first convolutional layer to be the number of principal convolutional channels, and sets the stride of the first convolutional layer to a value greater than 1. The feature compression module 43 sets the number of input channels of the second convolutional layer to be the number of principal convolutional channels, the number of output channels to a preset number of convolutional channels, and sets the stride of the first convolutional layer to 1. The feature compression module 43 sequentially uses the first and second convolutional layers to perform multiple convolutional processes on the equivalent image feature information to generate dimensionality-reduced image feature information.
[0079] The feature compression module 43 includes a first transformation processing layer and a first encoding layer; the feature decompression module 43 uses the first transformation processing layer to perform discrete cosine transform (DCT) processing on the feature information of the dimension-reduced image, and uses the first encoding layer to perform compression encoding processing on the feature information of the dimension-reduced image after DCT processing, to generate image compressed data.
[0080] The feature decompression module 44 performs inverse encoding and inverse transform processing on the image compressed data to generate dimensionality-reduced image feature information. The feature decompression module 44 uses the feature decompression module and performs multiple deconvolution processes on the dimensionality-reduced image feature information based on the main convolution channel number and a preset convolution channel number to generate equivalent image feature information corresponding to the dimensionality-reduced image feature information.
[0081] In one embodiment, the feature decompression module 44 includes a second encoding layer and a second transformation processing layer; the feature decompression module 44 uses the second encoding layer to perform inverse compression encoding processing on the image compressed data, and uses the second transformation processing layer to perform inverse DCT processing on the image compressed data after inverse compression encoding processing to generate dimension-reduced image feature information.
[0082] The feature decompression module 44 sets the number of input channels of the third convolutional layer to a preset number of convolutional channels and the number of output channels to the number of primary convolutional channels, and sets the stride of the third convolutional layer to 1. The feature decompression module 44 sets the number of input channels and output channels of the fourth convolutional layer to the number of primary convolutional channels, and sets the stride of the fourth convolutional layer to a value greater than 1. The feature decompression module 44 sequentially uses the third and fourth convolutional layers to perform multiple deconvolution processes on the equivalent image feature information to generate dimensionality-reduced image feature information.
[0083] In one embodiment, such as Figure 5As shown, the feature extraction module 46 performs feature extraction processing on the image to obtain multiple image feature information corresponding to the image; the feature extraction module 46 includes models such as the Feature Pyramid Network (FPN) model; the feature extraction module 46 uses the FPN model to process the image input and generate multiple image feature information; wherein, the image width dimension and image height dimension of the multiple image feature information are smaller than the corresponding image width dimension and image height dimension in the image, respectively; and the image width dimension and image height dimension of the multiple image feature information decrease sequentially.
[0084] Figure 6 This is a schematic diagram of a module according to yet another embodiment of the image processing apparatus of the present disclosure. Figure 6 As shown, the image processing apparatus may include a memory 61, a processor 62, a communication interface 63, and a bus 64. The memory 61 is used to store instructions, and the processor 62 is coupled to the memory 61. The processor 62 is configured to execute the image processing method described above based on the instructions stored in the memory 61.
[0085] The memory 61 can be a high-speed RAM, non-volatile memory, or a memory array. The memory 61 may also be divided into blocks, and these blocks can be combined into virtual volumes according to certain rules. The processor 62 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the image processing method of this disclosure.
[0086] In one embodiment, this disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the image processing method as described in any of the above embodiments.
[0087] The image processing method, apparatus, and storage medium in the above embodiments reduce computational complexity by eliminating convolution operations in feature stitching and feature segmentation and reconstruction; reduce lost information by eliminating GAP operations, and improve processing accuracy by preventing multi-level features from being recovered through other features; improve compression ratio by using convolution processing to reduce data volume and further compress channels; effectively reduce memory usage by eliminating FC operations and reducing convolution processing; and improve the user experience.
[0088] The methods and systems of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.
[0089] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.
Claims
1. An image processing method, comprising: Multiple image feature information is stitched together without convolution or modifying the length of the multiple image feature information to generate equivalent image feature information; the equivalent number of channels corresponding to the equivalent image feature information is determined, and the number of main convolution channels corresponding to the feature compression module is determined based on the equivalent number of channels; The feature compression module is used to perform multiple convolution processes on the equivalent image feature information based on the number of main convolution channels and the preset number of convolution channels to generate dimensionality-reduced image feature information corresponding to the equivalent image feature information. The feature compression module is used to transform and encode the feature information of the dimensionality-reduced image to generate compressed image data. The feature decompression module performs reverse processing on the image compressed data, corresponding to the processing performed by the feature compression module, to obtain the equivalent image feature information; The equivalent image feature information is segmented to obtain the multiple image feature information.
2. The method as described in claim 1, wherein the dimensions of the image feature information include: Number of channels, image width, and image height; Determining the equivalent number of channels corresponding to the equivalent image feature information includes: Obtain the minimum image width and minimum height among the dimensions of the multiple image feature information, and use them as the equivalent image width and equivalent image height among the dimensions of the equivalent image feature information; Calculate the product of the equivalent image width and the equivalent image height, and use the quotient of the total amount of data of the equivalent image feature information and the product as the equivalent number of channels.
3. The method as described in claim 2, wherein determining the number of main convolutional channels corresponding to the feature compression module based on the equivalent number of channels includes: Set the number of main convolution channels to the equivalent number of channels.
4. The method of claim 1, wherein, The feature compression module includes multiple first convolutional layers and at least one second convolutional layer; the step of using the feature compression module and performing multiple convolutional processes on the equivalent image feature information based on the main convolutional channel number and a preset convolutional channel number to generate dimensionality-reduced image feature information corresponding to the equivalent image feature information includes: Set the number of input channels and output channels of the first convolutional layer to the number of main convolutional channels, and set the stride of the first convolutional layer to a value greater than 1; The number of input channels of the second convolutional layer is set to the number of main convolutional channels, and the number of output channels is set to the preset number of convolutional channels. The stride of the second convolutional layer is set to 1. The preset number of convolutional channels includes 8, 16, and 32. The equivalent image feature information is processed by performing convolution multiple times using the first convolutional layer and the second convolutional layer in sequence to generate the dimensionality-reduced image feature information.
5. The method of claim 1, wherein the feature compression module comprises: A first transformation processing layer and a first encoding layer; the step of using the feature compression module to perform transformation and encoding processing on the dimensionality-reduced image feature information to generate image compressed data includes: The first transformation processing layer is used to perform Discrete Cosine Transform (DCT) processing on the dimensionality-reduced image feature information, and the first encoding layer is used to perform compression encoding processing on the dimensionality-reduced image feature information after DCT processing to generate the image compressed data.
6. The method of claim 1, wherein performing reverse processing on the image compressed data using the feature decompression module, corresponding to the processing performed by the feature compression module, comprises: The feature decompression module is used to perform inverse encoding and inverse transformation on the image compressed data to generate dimension-reduced image feature information. The feature decompression module is used to perform multiple deconvolution processes on the dimensionality-reduced image feature information based on the number of main convolution channels and the preset number of convolution channels, thereby generating equivalent image feature information corresponding to the dimensionality-reduced image feature information.
7. The method of claim 6, wherein the feature decompression module comprises: Second coding layer and second transform processing layer; The step of using the feature decompression module to perform inverse encoding and inverse transform processing on the image compressed data to generate dimensionality-reduced image feature information includes: The image compressed data is decompressed using the second encoding layer, and the image compressed data after decompression is processed by the second transform processing layer is then processed by inverse DCT to generate the dimensionality-reduced image feature information.
8. The method of claim 6, wherein, The feature decompression module includes at least one third convolutional layer and multiple fourth convolutional layers; using the feature decompression module and performing multiple deconvolution processes on the dimensionality-reduced image feature information based on the number of main convolutional channels and a preset number of convolutional channels includes: The number of input channels of the third convolutional layer is set to the preset number of convolutional channels, the number of output channels is set to the number of main convolutional channels, and the stride of the third convolutional layer is set to 1. The number of input channels and the number of output channels of the fourth convolutional layer are set to the number of channels of the main convolution, and the stride of the fourth convolutional layer is set to a value greater than 1. The third and fourth convolutional layers are used sequentially to perform multiple deconvolution processes on the dimensionality-reduced image feature information to generate the equivalent image feature information.
9. The method of claim 1, further comprising: The feature extraction module is used to perform feature extraction processing on the image to obtain multiple image feature information corresponding to the image.
10. The method of claim 9, wherein the feature extraction module comprises: The Feature Pyramid Network (FPN) model; the step of using a feature extraction module to perform feature extraction processing on the image and obtain multiple image feature information corresponding to the image includes: The image input is processed using the FPN model to generate the multiple image feature information; Wherein, the image width dimension and image height dimension of the plurality of image feature information are both smaller than the corresponding image width dimension and image height dimension in the image, respectively; and the image width dimension and image height dimension of the plurality of image feature information decrease sequentially.
11. An image processing apparatus, comprising: The feature stitching module is used to stitch together multiple image feature information without performing convolution processing on the multiple image feature information or modifying the length of the multiple image feature information, thereby generating equivalent image feature information. The channel number determination module is used to determine the equivalent channel number corresponding to the equivalent image feature information, and to determine the main convolution channel number corresponding to the feature compression module based on the equivalent channel number; The feature compression module is used to perform multiple convolution processes on the equivalent image feature information based on the main convolution channel number and the preset convolution channel number to generate dimensionality-reduced image feature information corresponding to the equivalent image feature information; and to use the feature compression module to perform transformation and encoding processes on the dimensionality-reduced image feature information to generate image compressed data. The feature decompression module is used to perform reverse processing on the image compressed data, corresponding to the processing performed by the feature compression module, to obtain the equivalent image feature information; The feature segmentation module is used to segment the equivalent image feature information to obtain the multiple image feature information.
12. The apparatus of claim 11, wherein the dimensions of the image feature information include: Number of channels, image width, and image height; The channel number determination module is used to obtain the minimum image width and minimum height in the dimensions of the multiple image feature information, as the equivalent image width and equivalent image height in the dimensions of the equivalent image feature information; calculate the product of the equivalent image width and the equivalent image height; and take the quotient of the total data volume of the equivalent image feature information and the product as the equivalent channel number.
13. The apparatus of claim 12, wherein, The channel number determination module is used to set the main convolution channel number to the equivalent channel number.
14. The apparatus of claim 11, wherein, The feature compression module includes multiple first convolutional layers and at least one second convolutional layer; The feature compression module is used to set the number of input channels and the number of output channels of the first convolutional layer to the number of main convolutional channels, and to set the stride of the first convolutional layer to a value greater than 1. The number of input channels of the second convolutional layer is set to the number of main convolutional channels, and the number of output channels is set to the preset number of convolutional channels. The stride of the second convolutional layer is set to 1. The preset number of convolutional channels includes 8, 16, and 32. The first convolutional layer and the second convolutional layer are used sequentially to perform multiple convolutional processes on the equivalent image feature information to generate the dimensionality-reduced image feature information.
15. The apparatus of claim 11, wherein the feature compression module comprises: First transformation processing layer and first coding layer; The feature compression module is used to perform Discrete Cosine Transform (DCT) processing on the dimensionality-reduced image feature information using the first transformation processing layer, and to perform compression encoding processing on the dimensionality-reduced image feature information after DCT processing using the first encoding layer, thereby generating the image compressed data.
16. The apparatus of claim 11, wherein, The feature decompression module is used to perform inverse encoding and inverse transformation processing on the image compressed data to generate dimensionality-reduced image feature information; and to perform multiple deconvolution processes on the dimensionality-reduced image feature information based on the number of main convolution channels and a preset number of convolution channels to generate equivalent image feature information corresponding to the dimensionality-reduced image feature information.
17. The apparatus of claim 16, wherein the feature decompression module comprises: Second coding layer and second transform processing layer; The feature decompression module is used to perform inverse compression encoding on the image compressed data using the second encoding layer, and to perform inverse DCT processing on the image compressed data after inverse compression encoding using the second transform processing layer, thereby generating the dimensionality-reduced image feature information.
18. The apparatus of claim 16, wherein, The feature decompression module includes at least one third convolutional layer and multiple fourth convolutional layers; The feature decompression module is used to set the number of input channels of the third convolutional layer to the preset number of convolutional channels and the number of output channels to the number of main convolutional channels, and to set the stride of the third convolutional layer to 1; set the number of input channels and the number of output channels of the fourth convolutional layer to the number of main convolutional channels, and to set the stride of the fourth convolutional layer to a value greater than 1; and sequentially use the third convolutional layer and the fourth convolutional layer to perform multiple deconvolution processes on the dimensionality-reduced image feature information to generate the equivalent image feature information.
19. The apparatus of claim 11, further comprising: The feature extraction module is used to perform feature extraction processing on the image to obtain multiple image feature information corresponding to the image.
20. The apparatus of claim 19, wherein the feature extraction module comprises: Feature Pyramid Network (FPN) model; The feature extraction module is used to process the image input using the FPN model to generate the plurality of image feature information; wherein the image width dimension and image height dimension of the plurality of image feature information are both smaller than the corresponding image width dimension and image height dimension in the image; and the image width dimension and image height dimension of the plurality of image feature information decrease sequentially.
21. An image processing apparatus, comprising: Memory; And a processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1 to 10 based on instructions stored in the memory.
22. A computer-readable storage medium that non-transitoryly stores computer instructions, which are executed by a processor according to any one of claims 1 to 10.