Image processing method and related device

Abstract

To provide an image processing method with which compression bit rate control is realized in the same compression model, and an image processing device.SOLUTION: An image processing method includes steps of: acquiring a first image; carrying out feature extraction on the first image to obtain at least one first feature map including N first feature values (N is an integer); acquiring a target compression bit rate, which corresponds to M target gain values, where each target gain value corresponds to one first feature value (M is a positive integer smaller than or equal to N); processing the corresponding first feature values according to the M target gain values to obtain M second feature values; and performing quantization and entropy coding on the first feature map including the M second feature values to obtain coded data.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to Chinese Patent Application No. 202010082808.4, entitled “IMAGE PROCESSING METHOD AND RELATED DEVICE”, filed with the State Intellectual Property Office of the People's Republic of China on February 7, 2020, which is hereby incorporated by reference in its entirety.

[0002] The present application relates to the field of artificial intelligence, and in particular to image processing methods and related apparatus. [Background technology]

[0003] Today, multimedia data accounts for a large proportion of Internet traffic. Image data compression plays a vital role in the storage and efficient transmission of multimedia data, making image coding a technology of great practical value.

[0004] Image coding has been researched for a long time. Researchers have proposed a large number of methods and formulated various international standards such as JPEG, JPEG2000, WebP, and BPG. Although these coding methods are now widely applied, these traditional methods show some limitations for the increasing amount of image data and the constantly emerging new media types.

[0005] In recent years, researchers have begun to study image coding methods based on deep learning. Some researchers have already achieved good results. For example, Balle et al. proposed an end-to-end optimal image coding method that surpasses the current best image coding performance and even surpasses the current best traditional coding standard BPG. However, currently, most image coding methods based on deep convolutional networks have a drawback, namely, one trained model can only output one coding result for one type of input image, so that the coding effect of the target compression bit rate based on the actual requirements cannot be obtained. Summary of the Invention [Means for solving the problem]

[0006] The present application provides an image processing method for implementing compression bit rate control in a uniform compression model.

[0007] According to a first aspect, the present application provides an image processing method, the method comprising: The method includes the steps of: obtaining a first image; performing feature extraction on the first image to obtain at least one first feature map, where the at least one first feature map includes N first feature values, where N is a positive integer; obtaining a target compression bit rate, where the target compression bit rate corresponds to M target gain values, where each target gain value corresponds to one first feature value, where M is a positive integer equal to or less than N; processing the corresponding first feature values ​​according to the M target gain values ​​to obtain M second feature values, where the original first feature map may be replaced with the at least one processed first feature map in this embodiment of the present application; and performing quantization and entropy coding on the at least one processed first feature map to obtain encoded data, where the at least one processed first feature map includes M second feature values. In the above scheme, different target gain values ​​are set for different target compression bit rates to implement compression bit rate control.

[0008] In an optional design of the first aspect, information entropy of the quantized data obtained by quantizing at least one processed first feature map satisfies a predetermined condition, the predetermined condition being related to a target compression bit rate.

[0009] In an optional design of the first embodiment, a higher target compression bit rate indicates a higher information entropy of the quantized data.

[0010] In an optional design of the first aspect, a difference between the compression bit rate corresponding to the encoded data and the target compression bit rate falls within a preset range.

[0011] In an optional design of the first aspect, the M second feature values ​​are obtained by performing multiplication operations respectively on the M target gain values ​​and the corresponding first feature values.

[0012] In an optional design of the first aspect, the at least one first feature map includes a first target feature map, the first target feature map including P first feature values, all of the P first feature values ​​corresponding to the same target gain value, where P is a positive integer less than or equal to M.

[0013] In an optional design of the first aspect, the method comprises: determining M target gain values ​​corresponding to a target compression bit rate based on a target mapping relationship, the target mapping relationship being used to indicate a correlation between the compression bit rate and the M target gain values; the target mapping relationship includes a plurality of compressed bit rates, a plurality of gain vectors, and a correlation between the plurality of compressed bit rates and the plurality of gain vectors, the target compressed bit rate being one of the plurality of compressed bit rates, and the M target gain values ​​being one element of the plurality of gain vectors; or If the target mapping relationship comprises an objective function mapping relationship, and the input of the objective function relationship comprises the target compression bit rate, then the output of the objective function relationship comprises the M target gain values.

[0014] In an optional design of the first aspect, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponds to M first gain values, the second compression bit rate corresponds to M second gain values, and the M target gain values ​​are obtained by performing an interpolation operation on the M first gain values ​​and the M second gain values.

[0015] In an optional design of the first aspect, the M first gain values ​​include a first target gain value, the M second gain values ​​include a second target gain value, and the M target gain values ​​include a third target gain value, where the first target gain value, the second target gain value, and the third target gain value correspond to the same feature value among the M first feature values, and the third target gain value is obtained by performing an interpolation operation on the first target gain value and the second target gain value.

[0016] In an optional design of the first aspect, the first image includes a target object, and the M first feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0017] In an optional design of the first aspect, each of the M target gain values ​​corresponds to one inverse gain value, and the inverse gain value is used to process feature values ​​obtained in a decoding process of the encoded data, and a product of each of the M target gain values ​​and the corresponding inverse gain value falls within a preset range.

[0018] In an optional design of the first aspect, the method further includes performing entropy decoding on the encoded data to obtain at least one second feature map, where the at least one second feature map includes N third feature values, each third feature value corresponding to one first feature value; obtaining M target inverse gain values, where each target inverse gain value corresponds to one third feature value; performing gain processing on corresponding third feature values ​​based on the M target inverse gain values ​​to obtain M fourth feature values, respectively; and performing image reconstruction on the at least one second feature map obtained after the inverse gain processing to obtain a second image, where the at least one second feature map obtained after the inverse gain processing includes M fourth feature values.

[0019] In an optional design of the first aspect, the M fourth feature values ​​are obtained by performing multiplication operations respectively on the M target inverse gain values ​​and the corresponding third feature values.

[0020] In an optional design of the first aspect, the at least one second feature map includes a second target feature map, the second target feature map including P third feature values, all of the P third feature values ​​corresponding to the same target inverse gain value, where P is a positive integer less than or equal to M.

[0021] In an optional design of the first aspect, the method further includes determining M target inverse gain values ​​corresponding to the target compression bitrates based on a target mapping relationship, where the target mapping relationship is used to indicate a correlation between the compression bitrates and the inverse gain vectors.

[0022] In an optional design of the first aspect, the target mapping relationship includes a plurality of compression bit rates, a plurality of inverse gain vectors, and a correlation relationship between the plurality of compression bit rates and the plurality of inverse gain vectors, where the target compression bit rate is one of the plurality of compression bit rates, and the M target inverse gain values ​​are elements of one of the plurality of inverse gain vectors.

[0023] In an optional design of the first aspect, the target mapping relationship comprises an objective function mapping relationship, where an input of the objective function relationship comprises the target compression bitrate, and an output of the objective function relationship comprises the M target inverse gain values.

[0024] In an optional design of the first aspect, the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0025] In an optional design of the first embodiment, the product of each of the M target gain values ​​and the corresponding target inverse gain value falls within a preset range.

[0026] In an optional design of the first aspect, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponds to the M first inverse gain values, the second compression bit rate corresponds to the M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values.

[0027] In an optional design of the first aspect, the M first inverse gain values ​​include a first target inverse gain value, the M second inverse gain values ​​include a second target inverse gain value, and the M target inverse gain values ​​include a third target inverse gain value, where the first target inverse gain value, the second target inverse gain value, and the third target inverse gain value correspond to the same feature value among the M third feature values, and the third target inverse gain value is obtained by performing an interpolation operation on the first target inverse gain value and the second target inverse gain value.

[0028] According to a second aspect, the present application provides an image processing method, the method comprising: The method includes the steps of: obtaining encoded data; performing entropy decoding on the encoded data to obtain at least one second feature map, where the at least one second feature map includes N third feature values, where N is a positive integer; obtaining M target inverse gain values, where each target inverse gain value corresponds to one third feature value, where M is a positive integer less than or equal to N; processing corresponding third feature values ​​based on the M target inverse gain values ​​to obtain M fourth feature values, respectively; and performing image reconstruction based on the at least one processed second feature map to obtain a second image, where the at least one processed second feature map includes the M fourth feature values.

[0029] In an optional design of the second aspect, the M fourth feature values ​​are obtained by performing multiplication operations respectively on the M target inverse gain values ​​and the corresponding third feature values.

[0030] In an optional design of the second aspect, the at least one second feature map includes a second target feature map, the second target feature map including P third feature values, all of the P third feature values ​​corresponding to the same target inverse gain value, where P is a positive integer less than or equal to M.

[0031] In an optional design of the second aspect, the method comprises: obtaining a target compression bit rate; determining M target inverse gain values ​​corresponding to the target compression bit rate based on a target mapping relationship, the target mapping relationship being used to indicate a correlation between the compression bit rate and the inverse gain vector; the target mapping relationship includes a plurality of compression bit rates, a plurality of inverse gain vectors, and a correlation between the plurality of compression bit rates and the plurality of inverse gain vectors; the target compression bit rate is one of a plurality of compression bit rates and the M target inverse gain values ​​are elements of a plurality of inverse gain vectors; or If the target mapping relationship comprises an objective function mapping relationship, and the input of the objective function relationship comprises the target compression bit rate, then the output of the objective function relationship comprises the M target inverse gain values.

[0032] In an optional design of the second aspect, the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0033] In an optional design of the second aspect, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponds to the M first inverse gain values, the second compression bit rate corresponds to the M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values.

[0034] In an optional design of the second aspect, the M first inverse gain values ​​include a first target inverse gain value, the M second inverse gain values ​​include a second target inverse gain value, and the M target inverse gain values ​​include a third target inverse gain value, where the first target inverse gain value, the second target inverse gain value, and the third target inverse gain value correspond to the same feature value among the M first feature values, and the third target inverse gain value is obtained by performing an interpolation operation on the first target inverse gain value and the second target inverse gain value.

[0035] According to a third aspect, the present application provides an image processing method, the method comprising: acquiring a first image; performing feature extraction on the first image based on the encoding network to obtain at least one first feature map, the at least one first feature map including N first feature values, where N is a positive integer; obtaining a target compression bit rate, the target compression bit rate corresponding to M initial gain values ​​and M initial inverse gain values, each initial gain value corresponding to one first feature value, each initial inverse gain value corresponding to one third feature value, M being a positive integer less than or equal to N; processing each of the corresponding first feature values ​​based on the M initial gain values ​​to obtain M second feature values; performing quantization and entropy coding on the at least one processed first feature map based on a quantization network and an entropy coding network to obtain coded data and a bit rate loss, where the at least one first feature map obtained after gain processing includes M second feature values; performing entropy decoding on the encoded data based on the entropy decoding network to obtain at least one second feature map, where the at least one second feature map includes M third feature values, each third feature value corresponding to one first feature value; processing the corresponding third feature values ​​according to the M initial inverse gain values ​​to obtain M fourth feature values, respectively; performing image reconstruction on at least one processed second feature map based on the decoding network to obtain a second image, where the at least one processed feature map includes M fourth feature values; obtaining a distortion loss of a second image relative to a first image; performing joint training on a first encoding / decoding network, M initial gain values, and M initial inverse gain values ​​by using a loss function until an image distortion value between a first image and a second image reaches a first preset degree, where the image distortion value is related to a bitrate loss and a distortion loss, and the encoding / decoding network includes an encoding network, a quantization network, an entropy encoding network, and an entropy decoding network; outputting a second encoding / decoding network, M target gain values, and M target inverse gain values, where the second encoding / decoding network is a model obtained after iterative training is performed on the first encoding / decoding network, and the M target gain values ​​and the M target inverse gain values ​​are obtained after iterative training is performed on the M initial gain values ​​and the M initial inverse gain values; Includes.

[0036] In an optional design of the third aspect, information entropy of the quantized data obtained by quantizing at least one first feature map obtained after gain processing satisfies a predetermined condition, and the predetermined condition is related to a target compression bit rate.

[0037] In an optional design of the third aspect, the preset condition at least includes indicating that the higher the target compression bit rate, the greater the information entropy of the quantized data.

[0038] In an optional design of the third aspect, the M second feature values ​​are obtained by performing multiplication operations respectively on the M target gain values ​​and the corresponding first feature values.

[0039] In an optional design of the third aspect, the at least one first feature map includes a first target feature map, the first target feature map including P first feature values, all of the P first feature values ​​corresponding to the same target gain value, where P is a positive integer less than or equal to M.

[0040] In an optional design of the third aspect, the first image includes a target object, and the M first feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0041] In an optional design of the third aspect, the product of each of the M target gain values ​​and the corresponding target inverse gain value falls within a preset range, and the product of each of the M initial gain values ​​and the corresponding initial inverse gain value falls within a preset range.

[0042] According to a fourth aspect, the present application provides an image processing apparatus, comprising: an acquisition module configured to acquire a first image; a feature extraction module configured to perform feature extraction on the first image to obtain at least one first feature map, the at least one first feature map comprising N first feature values, where N is a positive integer; The obtaining module obtains a target compression bit rate, the target compression bit rate corresponding to M target gain values, each target gain value corresponding to one first feature value, where M is a positive integer less than or equal to N; a gain module configured to process each of the corresponding first feature values ​​based on the M target gain values ​​to obtain M second feature values; a quantization and entropy coding module configured to perform quantization and entropy coding on the at least one processed first feature map to obtain coded data, wherein the at least one processed first feature map includes M second feature values; Equipped with.

[0043] In an optional design of the fourth aspect, the information entropy of the quantized data obtained by quantizing at least one processed first feature map satisfies a predetermined condition, the predetermined condition being related to a target compression bit rate.

[0044] In an optional design of the fourth aspect, the preset conditions include at least: This includes indicating that the higher the target compression bit rate, the greater the information entropy of the quantized data.

[0045] In an optional design of the fourth aspect, the difference between the compression bit rate corresponding to the encoded data and the target compression bit rate falls within a preset range.

[0046] In an optional design of the fourth aspect, the M second feature values ​​are obtained by performing multiplication operations respectively on the M target gain values ​​and the corresponding first feature values.

[0047] In an optional design of the fourth aspect, the at least one first feature map includes a first target feature map, the first target feature map including P first feature values, all of the P first feature values ​​corresponding to the same target gain value, where P is a positive integer less than or equal to M.

[0048] In an optional design of the fourth aspect, the apparatus comprises: The method further comprises: a determining module configured to determine M target gain values ​​corresponding to the target compression bitrate based on a target mapping relationship, the target mapping relationship being used to indicate a correlation between the compression bitrate and the M target gain values; the target mapping relationship includes a plurality of compressed bit rates, a plurality of gain vectors, and a correlation between the plurality of compressed bit rates and the plurality of gain vectors, the target compressed bit rate being one of the plurality of compressed bit rates, and the M target gain values ​​being one element of the plurality of gain vectors; or If the target mapping relationship comprises an objective function mapping relationship, and the input of the objective function relationship comprises the target compression bit rate, then the output of the objective function relationship comprises the M target gain values.

[0049] In an optional design of the fourth aspect, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponds to M first gain values, the second compression bit rate corresponds to M second gain values, and the M target gain values ​​are obtained by performing an interpolation operation on the M first gain values ​​and the M second gain values.

[0050] In an optional design of the fourth aspect, the M first gain values ​​include a first target gain value, the M second gain values ​​include a second target gain value, and the M target gain values ​​include a third target gain value, where the first target gain value, the second target gain value, and the third target gain value correspond to the same feature value among the M first feature values, and the third target gain value is obtained by performing an interpolation operation on the first target gain value and the second target gain value.

[0051] In an optional design of the fourth aspect, the first image includes a target object, and the M first feature values ​​are feature values ​​in at least one feature map that correspond to the target object.

[0052] In an optional design of the fourth aspect, each of the M target gain values ​​corresponds to one inverse gain value, and the inverse gain value is used to process feature values ​​obtained in a decoding process of encoded data, and a product of each of the M target gain values ​​and the corresponding inverse gain value falls within a preset range.

[0053] In an optional design of the fourth aspect, the apparatus comprises: a decoding module configured to perform entropy decoding on the encoded data to obtain at least one second feature map, the at least one second feature map including N third feature values, each third feature value corresponding to one first feature value; The obtaining module is further configured to obtain M target inverse gain values, each target inverse gain value corresponding to one third feature value; This device is an inverse gain module configured to perform gain processing on corresponding third feature values ​​based on the M target inverse gain values, respectively, to obtain M fourth feature values; a reconstruction module configured to perform image reconstruction on at least one second feature map obtained after the inverse gain processing to obtain a second image, wherein the at least one second feature map obtained after the inverse gain processing includes M fourth feature values; It further comprises:

[0054] In an optional design of the fourth aspect, the M fourth feature values ​​are obtained by performing multiplication operations respectively on the M target inverse gain values ​​and the corresponding third feature values.

[0055] In an optional design of the fourth aspect, the at least one second feature map includes a second target feature map, the second target feature map including P third feature values, all of the P third feature values ​​corresponding to the same target inverse gain value, where P is a positive integer less than or equal to M.

[0056] In the optical design of the fourth aspect, the determination module is The method is further configured to determine M target inverse gain values ​​corresponding to the target compression bitrates based on the target mapping relationship, the target mapping relationship being used to indicate a correlation between the compression bitrates and the inverse gain vectors.

[0057] In an optional design of the fourth aspect, the target mapping relationship includes a plurality of compression bit rates, a plurality of inverse gain vectors, and a correlation relationship between the plurality of compression bit rates and the plurality of inverse gain vectors, where the target compression bit rate is one of the plurality of compression bit rates, and the M target inverse gain values ​​are elements of one of the plurality of inverse gain vectors.

[0058] In an optional design of the fourth aspect, the target mapping relationship comprises an objective function mapping relationship, where an input of the objective function relationship comprises the target compression bitrate, and an output of the objective function relationship comprises the M target inverse gain values.

[0059] In an optional design of the fourth aspect, the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0060] In an optional design of the fourth embodiment, the product of each of the M target gain values ​​and the corresponding target inverse gain value falls within a preset range.

[0061] In an optional design of the fourth aspect, the target compression bit rate is greater than a first compression bit rate and less than a second compression bit rate, the first compression bit rate corresponds to M first inverse gain values, the second compression bit rate corresponds to M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values.

[0062] In an optional design of the fourth aspect, the M first inverse gain values ​​include a first target inverse gain value, the M second inverse gain values ​​include a second target inverse gain value, and the M target inverse gain values ​​include a third target inverse gain value, where the first target inverse gain value, the second target inverse gain value, and the third target inverse gain value correspond to the same feature value among the M first feature values, and the third target inverse gain value is obtained by performing an interpolation operation on the first target inverse gain value and the second target inverse gain value.

[0063] According to a fifth aspect, the present application provides an image processing apparatus, comprising: a capture module configured to capture the encoded data; a decoding module configured to perform entropy decoding on the encoded data to obtain at least one second feature map, the at least one second feature map including N third feature values, where N is a positive integer; The obtaining module is further configured to obtain M target inverse gain values, each target inverse gain value corresponding to one third feature value, where M is a positive integer less than or equal to N; an inverse gain module configured to process each of the corresponding third feature values ​​based on the M target inverse gain values ​​to obtain M fourth feature values; a reconstruction module configured to perform image reconstruction on the at least one processed second feature map to obtain a second image, wherein the at least one processed second feature map includes M fourth feature values; Equipped with.

[0064] In an optional design of the fifth aspect, the M fourth feature values ​​are obtained by performing multiplication operations respectively on the M target inverse gain values ​​and the corresponding third feature values.

[0065] In an optional design of the fifth aspect, the at least one second feature map includes a second target feature map, the second target feature map including P third feature values, all of the P third feature values ​​corresponding to the same target inverse gain value, where P is a positive integer less than or equal to M.

[0066] In an optional design of the fifth aspect, the acquisition module is further configured to acquire a target compression bit rate; This device is A determining module determines M target inverse gain values ​​corresponding to a target compression bit rate based on a target mapping relationship, the target mapping relationship being used to indicate a correlation between the compression bit rate and the inverse gain vector; the target mapping relationship includes a plurality of compression bit rates, a plurality of inverse gain vectors, and a correlation between the plurality of compression bit rates and the plurality of inverse gain vectors, the target compression bit rate being one of the plurality of compression bit rates, and the M target inverse gain values ​​being one element of the plurality of inverse gain vectors; or If the target mapping relationship comprises an objective function mapping relationship, and the input of the objective function relationship comprises the target compression bit rate, then the output of the objective function relationship comprises the M target inverse gain values.

[0067] In an optional design of the fifth aspect, the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0068] In an optional design of the fifth aspect, the target compression bit rate is greater than a first compression bit rate and less than a second compression bit rate, the first compression bit rate corresponds to M first inverse gain values, the second compression bit rate corresponds to M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values.

[0069] In an optional design of the fifth aspect, the M first inverse gain values ​​include a first target inverse gain value, the M second inverse gain values ​​include a second target inverse gain value, and the M target inverse gain values ​​include a third target inverse gain value, where the first target inverse gain value, the second target inverse gain value, and the third target inverse gain value correspond to the same feature value among the M first feature values, and the third target inverse gain value is obtained by performing an interpolation operation on the first target inverse gain value and the second target inverse gain value.

[0070] According to a sixth aspect, the present application provides an image processing apparatus, comprising: an acquisition module configured to acquire a first image; a feature extraction module configured to perform feature extraction on the first image based on the encoding network to obtain at least one first feature map, the at least one first feature map including N first feature values, where N is a positive integer; The obtaining module is further configured to obtain a target compression bit rate, the target compression bit rate corresponding to M initial gain values ​​and M initial inverse gain values, each initial gain value corresponding to one first feature value, each initial inverse gain value corresponding to one third feature value, and M is a positive integer less than or equal to N; a gain module configured to process each of the corresponding first feature values ​​based on the M initial gain values ​​to obtain M second feature values; a quantization and entropy coding module configured to perform quantization and entropy coding on the at least one processed first feature map based on a quantization network and an entropy coding network to obtain coded data and a bit rate loss, where the at least one first feature map obtained after the gain processing includes M second feature values; a decoding module configured to perform entropy decoding on the encoded data based on the entropy decoding network to obtain at least one second feature map, the at least one second feature map including M third feature values, each third feature value corresponding to one first feature value; an inverse gain module configured to process corresponding third feature values, respectively, based on the M initial inverse gain values ​​to obtain M fourth feature values; and a reconstruction module configured to perform image reconstruction on the at least one processed second feature map based on the decoding network to obtain a second image, wherein the at least one processed feature map includes M fourth feature values; The acquisition module is further configured to acquire a distortion loss of the second image relative to the first image; a training module configured to perform joint training on a first encoding / decoding network, the M initial gain values, and the M initial inverse gain values ​​by using a loss function until an image distortion value between the first image and the second image reaches a first preset degree, the image distortion value being related to a bitrate loss and a distortion loss, and the encoding / decoding network includes an encoding network, a quantization network, an entropy encoding network, and an entropy decoding network; an output module configured to output a second encoding / decoding network, M target gain values, and M target inverse gain values, where the second encoding / decoding network is a model obtained after iterative training is performed on the first encoding / decoding network, and the M target gain values ​​and the M target inverse gain values ​​are obtained after iterative training is performed on the M initial gain values ​​and the M initial inverse gain values; Equipped with.

[0071] In an optional design of the sixth aspect, the information entropy of the quantized data obtained by quantizing at least one first feature map obtained after gain processing satisfies a predetermined condition, and the predetermined condition is related to a target compression bit rate.

[0072] In an optional design of the sixth aspect, the preset conditions include at least: This includes indicating that the higher the target compression bit rate, the greater the information entropy of the quantized data.

[0073] In an optional design of the sixth aspect, the M second feature values ​​are obtained by performing multiplication operations respectively on the M target gain values ​​and the corresponding first feature values.

[0074] In an optional design of the sixth aspect, the at least one first feature map includes a first target feature map, the first target feature map including P first feature values, all of the P first feature values ​​corresponding to the same target gain value, where P is a positive integer less than or equal to M.

[0075] In an optional design of the sixth aspect, the first image includes a target object, and the M first feature values ​​are feature values ​​in at least one feature map that correspond to the target object.

[0076] In an optional design of the sixth aspect, the product of each of the M target gain values ​​and the corresponding target inverse gain value falls within a preset range, and the product of each of the M initial gain values ​​and the corresponding initial inverse gain value falls within a preset range.

[0077] According to a seventh aspect, an embodiment of the present application provides an execution device. The execution device may include a memory, a processor, and a bus system. The memory is configured to store a program, and the processor is configured to execute the program in the memory, the program being: acquiring a first image; performing feature extraction on the first image to obtain at least one first feature map, the at least one first feature map including N first feature values, where N is a positive integer; obtaining a target compression bit rate, the target compression bit rate corresponding to M target gain values, each target gain value corresponding to one first feature value, where M is a positive integer less than or equal to N; processing each of the corresponding first feature values ​​based on the M target gain values ​​to obtain M second feature values; performing quantization and entropy coding on the at least one processed first feature map to obtain encoded data, wherein the at least one processed first feature map includes M second feature values; Includes.

[0078] In an optional design of the seventh aspect, the execution device is a virtual reality VR device, a mobile phone, a tablet computer, a notebook computer, a server, or an intelligent wearable device.

[0079] In the seventh aspect of the present application, the processor may be further configured to execute the steps of the first aspect or any possible implementation of the first aspect. For details, please refer to the first aspect. The details will not be described again here.

[0080] According to an eighth aspect, an embodiment of the present application provides an execution device. The execution device may include a memory, a processor, and a bus system. The memory is configured to store a program, and the processor is configured to execute the program in the memory, the program being: obtaining encoded data; performing entropy decoding on the encoded data to obtain at least one second feature map, the at least one second feature map including N third feature values, where N is a positive integer; obtaining M target inverse gain values, each target inverse gain value corresponding to one third feature value, where M is a positive integer less than or equal to N; processing the corresponding third feature values ​​based on the M target inverse gain values ​​to obtain M fourth feature values, respectively; performing image reconstruction on the at least one processed second feature map to obtain a second image, wherein the at least one processed second feature map includes M fourth feature values; Includes.

[0081] In an optional design of the eighth aspect, the execution device is a virtual reality VR device, a mobile phone, a tablet computer, a notebook computer, a server, or an intelligent wearable device.

[0082] In the eighth aspect of the present application, the processor may be further configured to execute the steps of the second aspect or any possible implementation of the second aspect. For details, please refer to the second aspect. The details will not be described again here.

[0083] According to a ninth aspect, an embodiment of the present application provides a training device. The training device may include a memory, a processor, and a bus system. The memory is configured to store a program, and the processor is configured to execute the program in the memory, the program comprising: acquiring a first image; performing feature extraction on the first image based on the encoding network to obtain at least one first feature map, the at least one first feature map including N first feature values, where N is a positive integer; obtaining a target compression bit rate, the target compression bit rate corresponding to M initial gain values ​​and M initial inverse gain values, each initial gain value corresponding to one first feature value, each initial inverse gain value corresponding to one third feature value, M being a positive integer less than or equal to N; processing each of the corresponding first feature values ​​based on the M initial gain values ​​to obtain M second feature values; performing quantization and entropy coding on the at least one processed first feature map based on a quantization network and an entropy coding network to obtain coded data and a bit rate loss, where the at least one first feature map obtained after gain processing includes M second feature values; performing entropy decoding on the encoded data based on the entropy decoding network to obtain at least one second feature map, where the at least one second feature map includes M third feature values, each third feature value corresponding to one first feature value; processing the corresponding third feature values ​​according to the M initial inverse gain values ​​to obtain M fourth feature values, respectively; performing image reconstruction on at least one processed second feature map based on the decoding network to obtain a second image, where the at least one processed feature map includes M fourth feature values; obtaining a distortion loss of a second image relative to a first image; performing joint training on a first encoding / decoding network, M initial gain values, and M initial inverse gain values ​​by using a loss function until an image distortion value between a first image and a second image reaches a first preset degree, where the image distortion value is related to a bitrate loss and a distortion loss, and the encoding / decoding network includes an encoding network, a quantization network, an entropy encoding network, and an entropy decoding network; outputting a second encoding / decoding network, M target gain values, and M target inverse gain values, where the second encoding / decoding network is a model obtained after iterative training is performed on the first encoding / decoding network, and the M target gain values ​​and the M target inverse gain values ​​are obtained after iterative training is performed on the M initial gain values ​​and the M initial inverse gain values; Includes.

[0084] In the ninth aspect of the present application, the processor may be further configured to execute the steps of the third aspect or any possible implementation of the third aspect. For details, please refer to the third aspect. The details will not be described again here.

[0085] According to a tenth aspect, an embodiment of the present application provides a computer readable storage medium storing a computer program which, when run on a computer, enables the computer to perform the image processing method according to any one of the first to third aspects.

[0086] According to an eleventh aspect, an embodiment of the present application provides a computer program which, when run on a computer, enables the computer to perform the image processing method according to any one of the first to third aspects.

[0087] According to a twelfth aspect, the present application provides a chip system. The chip system includes a processor configured to support the execution device or training device in performing the functions in the aforementioned aspects, for example, in transmitting or processing data and / or information in the aforementioned methods. In a possible design, the chip system further includes a memory. The memory is configured to store necessary program instructions and data of the execution device or training device. The chip system may include a chip, or may include a chip and another discrete component.

[0088] An embodiment of the present application provides an image processing method, in which a first image is obtained, feature extraction is performed on the first image to obtain at least one first feature map, the at least one first feature map includes N first feature values, N is a positive integer, a target compression bit rate is obtained, the target compression bit rate corresponds to M target gain values, each target gain value corresponds to one first feature value, M is a positive integer less than or equal to N, the corresponding first feature value is respectively processed based on the M target gain values ​​to obtain M second feature values, and quantization and entropy coding are performed on the at least one processed first feature map to obtain encoded data, the at least one processed first feature map includes M second feature values. In the above scheme, different target gain values ​​are set for different target compression bit rates to implement compression bit rate control. [Brief description of the drawings]

[0089] [Figure 1] 1 is a schematic diagram of the structure of the artificial intelligence main framework. [Figure 2a] FIG. 2 illustrates an application scenario according to an embodiment of the present application. [Figure 2b] FIG. 2 illustrates an application scenario according to an embodiment of the present application. [Diagram 3] FIG. 2 illustrates an embodiment of an image processing method according to an embodiment of the present application. [Figure 4] FIG. 1 illustrates a CNN-based image processing process. [Figure 5a] FIG. 13 illustrates information entropy distribution of feature maps at compressed bit rates according to an embodiment of the present application. [Figure 5b] FIG. 13 illustrates information entropy distribution of feature maps at compressed bit rates according to an embodiment of the present application. [Figure 6] FIG. 2 illustrates an objective function mapping relationship according to an embodiment of the present application. [Figure 7] FIG. 2 illustrates an embodiment of an image processing method according to an embodiment of the present application. [Figure 8] FIG. 2 illustrates an image compression procedure according to an embodiment of the present application. [Figure 9] FIG. 1 illustrates a compression effect according to an embodiment of the present application. [Figure 10] FIG. 2 illustrates a training process according to an embodiment of the present application. [Figure 11] FIG. 2 illustrates an image processing process according to an embodiment of the present application. [Figure 12] FIG. 1 illustrates a system architecture of an image processing system according to an embodiment of the present invention. [Figure 13] 1 is a schematic flow chart of an image processing method according to an embodiment of the present application. [Figure 14] 1 is a schematic diagram of the structure of an image processing device according to an embodiment of the present invention; [Figure 15] 1 is a schematic diagram of the structure of an image processing device according to an embodiment of the present invention; [Figure 16] 1 is a schematic diagram of the structure of an image processing device according to an embodiment of the present invention; [Figure 17] FIG. 2 is a schematic diagram of the structure of an execution device according to an embodiment of the present application; [Figure 18] 1 is a schematic diagram of a structure of a training device according to an embodiment of the present application. [Figure 19] FIG. 2 is a schematic diagram of the structure of a chip according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0090] Hereinafter, the embodiments of the present invention will be described with reference to the drawings in the embodiments of the present invention. The terms used in the embodiments of the present invention are merely used to describe specific embodiments of the present invention, and are not intended to limit the present invention.

[0091] Hereinafter, embodiments of the present application will be described with reference to the drawings. Those skilled in the art can learn that the technical solutions provided in the embodiments of the present application can also be applied to similar technical problems as technology evolves and new scenarios emerge.

[0092] In the specification, claims, and accompanying drawings of this application, the terms "first", "second", etc. are intended to distinguish between similar objects, but do not necessarily indicate a particular order or sequence. It should be understood that the terms used in this manner are interchangeable in appropriate circumstances, and this is merely a distinguishing method used in describing objects having the same attributes in the embodiments of this application. In addition, the terms "include", "have" and any other variations thereof are meant to cover non-exclusive inclusions, and thus a process, method, system, product, or device that includes a set of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to such a process, method, product, or device.

[0093] First, the overall operating procedure of the artificial intelligence system is described. Figure 1 is a schematic diagram of the structure of the artificial intelligence main framework. The following describes the aforementioned artificial intelligence main framework from two dimensions, namely, the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the overall process from data acquisition to data processing. For example, the process may be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, the data undergoes a condensation process of "data information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the infrastructure and information (technology provision and processing implementation) that underpins human intelligence, to the industrial ecological process of the system.

[0094] (1) Infrastructure The infrastructure provides computational support to the artificial intelligence system, communicates with the outside world, and performs the support by using the base platform. The infrastructure communicates with the outside world by using sensors. Computational capabilities are provided by intelligent chips (CPU, NPU, GPU, ASIC, or hardware acceleration chips such as FPGA). The base platform includes the assurance and support of related platforms such as distributed computing frameworks and networks, and can include cloud storage and computing, interconnection and interaction networks, etc. For example, sensors communicate with the outside world to obtain data, and the data is provided to the intelligent chips in the distributed computing system provided by the base platform for computation.

[0095] (2) Data Data from the higher layers of the infrastructure represent data sources in the field of artificial intelligence. The data relate to graphs, images, voice, text, and further relate to Internet of Things data of traditional devices, service data of existing systems, and sensory data such as force, displacement, liquid level, temperature, humidity, etc.

[0096] (3) Data processing Data processing typically includes methods such as data training, machine learning, deep learning, search, inference, and decision-making.

[0097] Machine learning and deep learning can refer to the codified and formalized intelligent information modeling, extraction, pre-processing, training, etc., performed on data.

[0098] Inference is the process of simulating intelligent human reasoning in a computer or intelligent system, using formalized information to perform machine thinking and solve problems based on inference control policies. Typical functions of inference are searching and matching.

[0099] Decision making is a process in which decisions are made after intelligent information reasoning, usually providing functions such as classification, ranking, and prediction.

[0100] (4) General Ability After the above-mentioned data processing is performed on the data, some general capabilities may be further formed based on the data processing results, such as algorithms or general systems, to perform translation, text analysis, computer vision processing, voice recognition, image recognition, etc.

[0101] (5) Intelligent products and industrial applications Intelligent products and industrial applications refer to the products and applications of artificial intelligence systems in various fields, which refer to the encapsulation of the entire artificial intelligence solution, the commercialization of intelligent information decision-making, and the realization of ground-based applications, whose application fields mainly include intelligent terminals, intelligent transportation, intelligent medical care, autonomous driving, safe cities, etc.

[0102] The present application may be applied to the image processing field in the field of artificial intelligence, and the following describes several application scenarios of the product landing.

[0103] I. Application to image compression processing in terminal devices The image compression method provided in the embodiment of the present application may be applied to the image compression process in a terminal device, specifically, to albums, video surveillance, etc. in a terminal device. For details, please refer to FIG. 2a. FIG. 2a illustrates an application scenario according to an embodiment of the present application. As shown in FIG. 2a, the terminal device can obtain a photo to be compressed. The photo to be compressed may be a photo taken by a camera or a frame of a photo extracted from a video. The terminal device can perform feature extraction on the obtained photo to be compressed by using an artificial intelligence (AI) encoding unit of an embedded neural network processing unit (NPU), convert the image data into output features with less redundancy, and generate probability estimates of points in the output features. The central processing unit (CPU) performs arithmetic coding on the extracted output features by using the probability estimates of each point in the output features, reducing the coding redundancy of the output features, further reducing the amount of data transmission in the image compression process, and stores the encoded data obtained after encoding in a corresponding storage location in the form of a data file. When a user needs to retrieve a file stored in a memory location, the CPU can retrieve and load the stored file from the corresponding memory location, obtain a decoded feature map based on arithmetic decoding, and perform reconstruction on the feature map by using the AI ​​decoding unit in the NPU to obtain a reconstructed image.

[0104] 2. Application to image compression processing on the cloud The image compression method provided in the embodiment of the present application may be applied to an image compression process on the cloud side, specifically, to a function such as a cloud album in a server on the cloud side. For details, please refer to FIG. 2b. FIG. 2b illustrates an application scenario according to an embodiment of the present application. As shown in FIG. 2b, a terminal device can obtain a photo to be compressed. The photo to be compressed may be a photo taken by a camera or a frame of a photo extracted from a video. The terminal device can perform lossless encoding compression on the photo to be compressed by using a CPU to obtain encoded data, and the lossless encoding compression may be performed, for example, based on any lossless compression method in the prior art, but is not limited to being performed based thereon. The terminal device can send the encoded data to a server on the cloud side. The server can perform corresponding lossless decoding on the received encoded data to obtain an image to be compressed. The server can perform feature extraction on the obtained photo to be compressed by using an AI encoding unit in a graphics processing unit (GPU) to convert image data into output features with less redundancy, and generate probability estimates of points in the output features. The CPU performs arithmetic coding on the extracted output features by using the probability estimates of the points in the output features to reduce the coding redundancy of the output features, and further reduces the amount of data transmission in the image compression process, and stores the coded data obtained after coding in the corresponding storage location in the form of a data file. When a user needs to retrieve the file stored in the storage location, the CPU retrieves and loads the stored file from the corresponding storage location, retrieves a decoded feature map based on arithmetic decoding, and performs reconstruction on the feature map by using the AI ​​decoding unit in the NPU to retrieve a reconstructed image.The server can perform lossless encoding compression on the photo to be compressed by using the CPU to obtain encoded data, and the lossless encoding compression can be performed, for example, but not limited to, based on any lossless compression method in the prior art. The server can transmit the encoded data to the terminal device, and the terminal device can perform corresponding lossless decoding on the received encoded data to obtain a decoded image.

[0105] In this embodiment of the present application, a step of performing gain processing on the feature values ​​in the feature map may be added between the AI ​​encoding unit and the quantization unit, and a step of performing inverse gain processing on the feature values ​​in the feature map may be added between the arithmetic decoding unit and the AI ​​decoding unit. Next, the image processing method in the embodiment of the present invention will be described in detail.

[0106] Since the embodiments of the present application relate to a large number of neural network applications, for ease of understanding, the following first describes the relevant terms and concepts of neural networks that may be used in the embodiments of the present application.

[0107] (1) Neural Networks The neural network may include neurons, which may be computational units that use xs and intercept 1 as inputs, and the output of the computational units may be:

number

[0108] (2) Deep Neural Networks A deep neural network (DNN), also called a multi-layer neural network, may be understood as a neural network with multiple hidden layers. DNNs are divided based on the location of different layers. The neural network inside a DNN is classified into three types: input layer, hidden layer, and output layer. In general, the first layer is the input layer, the last layer is the output layer, and the middle layer is a hidden layer. The layers are fully connected. Specifically, any neuron in the i-th layer is necessarily connected to any neuron in the (i+1)-th layer.

[0109] Although DNN looks very complicated, the function of each layer is not complicated. In other words, DNN is a linear relationship as follows:

number

number

number

number

number

number

number

number

[0110] In conclusion, the coefficient from the kth neuron in the (L-1)th layer to the jth neuron in the Lth layer is

number

[0111] Note that there is no parameter W in the input layer. In a deep neural network, the more hidden layers there are, the more capable the network is of describing complex cases in the real world. Theoretically, a model with more parameters is more complex and has a larger "capacity", which indicates that the model can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and the final goal of training is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by the vector W of multiple layers).

[0112] (3) Convolutional Neural Networks A convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network includes a feature extractor including a convolutional layer and a subsampling layer. The feature extractor can be regarded as a filter. The convolutional layer is a neuron layer that performs convolution processing on an input signal in a convolutional neural network. In a convolutional layer of a convolutional neural network, one neuron may be connected to only a part of neurons in an adjacent layer. A convolutional layer generally includes several feature planes, and each feature plane may include neurons arranged in several rectangles. Neurons in the same feature plane share weights, where the shared weights are convolution kernels. The weight sharing may be understood as a method of extracting image information that is position-independent. The convolutional kernels may be initialized in the form of a matrix of random size. In the training process of a convolutional neural network, appropriate weights may be obtained for the convolutional kernels through learning. In addition, weight sharing is advantageous because it reduces connections between layers of a convolutional neural network, reducing the risk of overfitting.

[0113] (4) Loss function In the process of training a deep neural network, the current network prediction value may be compared with the expected target value, since it is expected that the output of the deep neural network is maximally close to the actual predicted value, and then the weight vector in each layer of the neural network is updated based on the difference between the current prediction value and the target value (usually there is an initialization process before the first update, in other words, the parameters are preset for each layer of the deep neural network). For example, if the network prediction value is large, the weight vector is adjusted to lower the prediction value until the deep neural network can predict the actual expected target value or a value close to the actual expected target value. Therefore, "how to obtain the difference between the prediction value and the target value by comparison" needs to be defined in advance. This is the loss function or objective function. The loss function and objective function are important formulas used to measure the difference between the prediction value and the target value. The loss function is used as an example. The larger the output value (loss) of the loss function, the larger the difference. Therefore, training a deep neural network is a process of minimizing the loss as much as possible.

[0114] (5) Backpropagation Algorithm In the training process, the neural network can correct the value of the parameters in the initial neural network model by using the back propagation (BP) algorithm, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the input signal is forward forwarded until an error loss is generated at the output, and the parameters of the initial neural network model are updated based on the back propagation error loss information so that the error loss becomes smaller. The back propagation algorithm is a back propagation movement that mainly relies on the error loss, and aims to obtain the optimal neural network model parameters, such as the weight matrix.

[0115] The embodiments of the present application first provide an explanation by using an example in which the application scenario is a terminal device.

[0116] For example, the terminal device may be a mobile phone, a tablet computer, a notebook computer, or an intelligent wearable device, and the terminal device may perform compression processing on the captured photo. In another example, the terminal device may be a virtual reality (VR) device. As another example, the embodiment of the present application may also be applied to intelligent monitoring. In intelligent monitoring, a camera may be configured. In this case, in intelligent monitoring, a photo to be compressed, etc. may be acquired by using a camera. It should be understood that the embodiment of the present application may further be applied to other scenarios in which image compression needs to be performed. Other application scenarios are not listed one by one here.

[0117] 3 shows an embodiment of an image processing method according to an embodiment of the present application. As shown in FIG. 3, the image processing method provided in this embodiment of the present application includes the following steps:

[0118] 301. Acquire a first image.

[0119] In this embodiment of the present application, the first image is the image to be compressed. The first image may be the image taken by the aforementioned terminal device by using a camera, or the first image may be the image obtained from the terminal device (for example, the image stored in the album of the terminal device, or the photo obtained from the cloud by the terminal device). It should be understood that the first image may be the image having image compression requirements, and the source of the image to be processed is not limited in the present application.

[0120] 302. Perform feature extraction on the first image to obtain at least one first feature map, the at least one first feature map including N first feature values, where N is a positive integer.

[0121] In this embodiment of the present application, optionally, the terminal device can perform feature extraction on the first image based on CNN to obtain at least one first feature map. Hereinafter, the first feature map may also be referred to as a channel-wise feature map, and each semantic channel corresponds to one first feature map (channel-wise feature map).

[0122] In this embodiment of the present application, Fig. 4 shows a CNN-based image processing process. Fig. 4 shows a first image 401, a CNN 402, and a number of first feature maps 403. The CNN 402 can include a number of CNN layers.

[0123] For example, the CNN 402 may multiply the top left 3x3 pixels of the input data (first image) by a weight and map it to the top left neuron of the first feature map. The weight to be multiplied is also 3x3. Then, in a similar process, the CNN 402 scans the input data (first image) from left to right and top to bottom, multiplies the input data by a weight, and maps it to the neuron of the feature map. In this specification, the 3x3 weight used is called a filter or a filter core. That is, the process of applying a filter to the CNN 402 is a process of performing a convolution operation by using a filter core, and the extracted result is called a "first feature map". The first feature map may also be called a multi-channel-wise feature map, and the term "multi-channel-wise feature map" may be a set of feature maps corresponding to multiple channels. According to an embodiment, the multi-channel-wise feature map may be generated by the CNN 402, which is also called the "feature extraction layer" or "convolution layer" of the CNN. A layer of the CNN may define a mapping from an output to an input. The mapping defined by a layer is implemented as one or more filter cores (convolution cores) applied to the input data, producing a feature map that is output to the next layer. The input data may be an image or a feature mapping image for the particular layer.

[0124] See Figure 4. During forward execution, the CNN 402 receives a first image 401 and generates a multi-channel-wise feature map 403 as an output. In addition, during forward execution, the next layer 402 receives the multi-channel-wise feature map 403 as an input and generates the multi-channel-wise feature map 403 as an output. Then, each subsequent layer receives the multi-channel-wise feature map generated in the previous layer and generates the next multi-channel-wise feature map as an output. Finally, the multi-channel-wise feature map generated in the (N)th layer is received.

[0125] Additionally, other processing operations may be performed in addition to applying a convolution core to map the input feature map to an output feature map. Examples of other processing operations may include, but are not limited to, applying an activation function, pooling, resampling, etc.

[0126] It should be noted that the above is only one embodiment for performing feature extraction on the first image. In practical applications, the specific embodiment of feature extraction is not limited.

[0127] In the present embodiment of the present application, in the above-mentioned method, the original image (first image) is transformed into another space (at least one first feature map) by using a CNN convolutional neural network. Optionally, there are 192 feature maps, i.e., there are 192 semantic channels, and each semantic channel corresponds to one first feature map. In the present embodiment of the present application, the at least one first feature map may be in the form of a three-dimensional tensor, and the size of the tensor may be 192×w×h, where w×h is the width and length of the matrix corresponding to the first feature map of a single channel.

[0128] In this embodiment of the present application, feature extraction may be performed on the first image to obtain a plurality of feature values. At least one first feature map may include a part or all of the plurality of feature values. Gain processing may not be performed on feature maps corresponding to some semantic channels that have a relatively small effect on the compression result. In this case, the at least one first feature map includes a part of the plurality of feature values.

[0129] In this embodiment of the present application, the at least one first feature map includes N first feature values, where N is a positive integer.

[0130] 303. Obtain a target compression bit rate, where the target compression bit rate corresponds to M target gain values, each target gain value corresponds to one first feature value, and M is a positive integer less than or equal to N.

[0131] In this embodiment of the present application, the terminal device can obtain a target compression bit rate. The target compression bit rate may be specified by a user, or may be determined by the terminal device based on the first image. This is not limited here.

[0132] In this embodiment of the present application, the target compression bit rate corresponds to M target gain values, each target gain value corresponds to one first feature value, and M is a positive integer less than or equal to N. That is, there is a certain correlation between the target compression bit rate and the M target gain values, and after obtaining the target compression bit rate, the terminal device can determine the M corresponding target gain values ​​according to the obtained target compression bit rate.

[0133] Optionally, in one embodiment, the terminal device can determine M target gain values ​​corresponding to the target compression bitrate based on the target mapping relationship. The target mapping relationship is used to indicate the correlation between the compression bitrate and the M target gain values. The target mapping relationship may be a pre-stored mapping relationship. After obtaining the target compression bitrate, the terminal device can directly find the target mapping relationship corresponding to the target compression bitrate in the corresponding storage location.

[0134] Optionally, in one embodiment, the target mapping relationship may include a plurality of compressed bit rates, a plurality of gain vectors, and a correlation relationship between the plurality of compressed bit rates and the plurality of gain vectors, where the target compressed bit rate is one of the plurality of compressed bit rates, and the M target gain values ​​are elements of one of the plurality of gain vectors.

[0135] In this embodiment of the present application, the target mapping relationship may be a pre-set table or another form. The target mapping relationship includes a plurality of compression bit rates and a gain vector corresponding to the compression bit rates. The gain vector may include a plurality of elements, each compression bit rate corresponds to M target gain values, and the M target gain values ​​are elements included in the gain vector corresponding to each compression bit rate.

[0136] Optionally, in one embodiment, the target mapping relationship may include an objective function mapping relationship, where an input of the objective function relationship includes a target compression bitrate and an output of the objective function relationship includes M target gain values.

[0137] In this embodiment of the present application, the target mapping relationship may be a preset target function mapping relationship or another form. The target function mapping relationship can indicate at least a correspondence relationship between a compression bit rate and a gain value. When the input of the target function relationship includes a target compression bit rate, the output of the target function relationship includes M target gain values.

[0138] It should be noted that in this embodiment of the present application, some or all of the M target gain values ​​may be the same. In this case, a number less than M may be used to indicate a target gain value corresponding to a first feature value in the M target feature values. For example, in one embodiment, the at least one first feature map includes a first target feature map, the first target feature map includes P first feature values, all of the P first feature values ​​correspond to the same target gain value, and P is a positive integer less than or equal to M. That is, the P first feature values ​​are feature values ​​of the same semantic channel and correspond to the same target gain value. In this case, the P first feature values ​​may be indicated by using one gain value.

[0139] In another embodiment, if the gain values ​​of the first feature values ​​corresponding to each semantic channel are the same, the M first gain values ​​may be represented by using the same number of target gain values ​​as there are semantic channels. Specifically, if there are 192 semantic channels (first feature maps), the M first gain values ​​may be represented by using 192 gain values.

[0140] In this embodiment of the present application, the first feature values ​​included in each of all or a portion of the at least one first feature map can correspond to the same target gain value. In this case, the at least one first feature map includes a first target feature map, the first target feature map includes P first feature values, all of the P first feature values ​​correspond to the same target gain value, and P is a positive integer less than or equal to M. That is, the first target feature map is one of the at least one first feature map, includes P first feature values, and all of the P first feature values ​​correspond to the same target gain value.

[0141] In this embodiment of the present application, the N first feature values ​​may be all feature values ​​included in at least one first feature map. When M is equal to N, this corresponds to all feature values ​​included in at least one first feature map each having a corresponding target gain value. When M is less than N, this corresponds to a portion of feature values ​​included in at least one first feature map each having a corresponding target gain value. In one embodiment, when the number of first feature maps is greater than 1, all feature values ​​included in each of the portions of at least one first feature map each have a corresponding target gain value, and a portion of feature values ​​included in each of the portions of at least one first feature map each have a corresponding target gain value.

[0142] Optionally, in one embodiment, the first image includes a target object, and the M first feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0143] In this embodiment of the present application, in some scenarios, the M first feature values ​​are feature values ​​corresponding to one or more target objects among the N first feature values. For example, for video content captured on a monitor, gain processing may not be performed on areas where the scene is relatively stationary, and gain processing may be performed on the content of objects or people passing through the area.

[0144] 304. Process the corresponding first feature values ​​based on the M target gain values ​​respectively to obtain M second feature values.

[0145] In this embodiment of the present application, after the target compression bit rate and the M target gain values ​​corresponding to the target compression bit rate are obtained, the corresponding first feature values ​​may be processed based on the M target gain values ​​respectively to obtain M second feature values. In one embodiment, the M second feature values ​​may be obtained by performing multiplication operations on the M target gain values ​​and the corresponding first feature values ​​respectively, i.e., the corresponding second feature values ​​may be obtained after the first feature values ​​are multiplied by the corresponding target gain values.

[0146] In this embodiment of the present application, in order to implement the effect of different compression bit rates in the same AI compression model, different target gain values ​​may be obtained for different obtained target compression bit rates. After the corresponding first feature values ​​are respectively processed based on the M target gain values ​​to obtain M second feature values, the distribution of the N first feature values ​​included in the at least one feature map corresponding to the original first image changes due to the M first feature values ​​that are subjected to gain processing.

[0147] In this embodiment of the present application, FIG. 5a and FIG. 5b show the distribution of feature maps for different compression bit rates according to the embodiment of the present application. Different compression bit rates are represented by using different bits per pixel (bpp). bpp represents the number of bits used to store each pixel, and a smaller bpp indicates a smaller compression bit rate. FIG. 5a shows the distribution of N first feature values ​​when bpp is 1. FIG. 5b shows the distribution of N first feature values ​​when bpp is 0.15. The output features (N first feature values) of the encoding network of the model with a higher compression bit rate have a larger variance in the statistical histogram, and thus a larger information entropy is obtained after quantization. Therefore, provided that different compression bit rates correspond to different target gain values, gain processing is performed to different degrees on the N first feature values ​​based on different target compression bit rates, so that the reconstruction effect of multiple bit rates can be implemented in a single AI compression model. Specifically, the selection rules for M target gain values ​​are as follows: The larger the target compression bit rate, the more dispersed the distribution of the N first feature values ​​obtained after the corresponding first feature values ​​are processed respectively based on the M target gain values, and thus the larger the information entropy obtained after quantization.

[0148] In this embodiment of the present application, after feature extraction is performed on the first image, all extracted first feature maps need to be processed to obtain multiple first feature maps. The feature values ​​included in the multiple first feature maps correspond to the same target gain value. In this case, all feature values ​​included in the multiple first feature maps are multiplied by the corresponding target gain value to change the distribution of the N first feature values ​​included in the multiple first feature maps. A larger target compression bit rate indicates a more dispersed distribution of the N first feature values.

[0149] In this embodiment of the present application, after feature extraction is performed on the first image, all extracted first feature maps need to be processed to obtain a plurality of first feature maps. The feature values ​​included in each of the plurality of first feature maps correspond to the same target gain value, i.e., each first feature map corresponds to one target gain value. In this case, the feature values ​​included in each of the plurality of first feature maps are multiplied by the corresponding target gain value to change the distribution of the N first feature values ​​included in the plurality of first feature maps. A larger target compression bit rate indicates a more dispersed distribution of the N first feature values.

[0150] In this embodiment of the present application, after feature extraction is performed on the first image, all the extracted first feature maps need to be processed to obtain a plurality of first feature maps. The feature values ​​included in each of the parts of the first feature maps correspond to the same target gain value, and the feature values ​​included in each of the remaining parts of the first feature maps correspond to different target gain values, i.e., each of the parts of the first feature maps corresponds to one target gain value, and each of the remaining parts of the first feature maps corresponds to multiple target gain values ​​(different feature values ​​in the same feature map may correspond to different target gain values). In this case, the feature values ​​included in each of the parts of the multiple first feature maps are multiplied by the corresponding target gain value, and the feature values ​​included in the remaining parts of the multiple first feature maps are multiplied by the corresponding target gain value, in order to change the distribution of the N first feature values ​​included in the plurality of first feature maps. A larger target compression bit rate indicates a more dispersed distribution of the N first feature values.

[0151] In this embodiment of the present application, after feature extraction is performed on the first image to obtain a plurality of first feature maps, some of the extracted first feature maps need to be processed (gain processing may not be performed on the first feature maps corresponding to some semantic channels that have a relatively small effect on the compression result). The number of extracted first feature maps that need to be processed is greater than 1. The feature values ​​included in each of the plurality of first feature maps correspond to the same target gain value, i.e., each first feature map corresponds to one target gain value. In this case, the feature values ​​included in each of the plurality of first feature maps are multiplied by the corresponding target gain value to change the distribution of the N first feature values ​​included in the plurality of first feature maps. A larger target compression bit rate indicates a more dispersed distribution of the N first feature values.

[0152] In this embodiment of the present application, after feature extraction is performed on the first image to obtain a plurality of first feature maps, a portion of the extracted first feature maps needs to be processed (gain processing may not be performed on the first feature maps corresponding to some semantic channels that have a relatively small effect on the compression result). The number of extracted first feature maps that need to be processed is greater than 1. The feature values ​​included in each of the portion of the first feature maps correspond to the same target gain value, and the feature values ​​included in each of the remaining portions of the first feature maps correspond to different target gain values, i.e., each of the portion of the first feature maps corresponds to one target gain value, and each of the remaining portions of the first feature maps corresponds to multiple target gain values ​​(different feature values ​​in the same feature map may correspond to different target gain values). In this case, in order to change the distribution of the N first feature values ​​included in the plurality of first feature maps, the feature values ​​included in each of the portion of the plurality of first feature maps are multiplied by the corresponding target gain value, and the feature values ​​included in the remaining portions of the plurality of first feature maps are multiplied by the corresponding target gain value. A larger target compression bit rate indicates a more dispersed distribution of the N first feature values.

[0153] In this embodiment of the present application, after feature extraction is performed on the first image to obtain multiple first feature maps, some of the extracted first feature maps need to be processed (gain processing may not be performed on the first feature maps corresponding to some semantic channels that have a relatively small effect on the compression result). The number of extracted first feature maps that need to be processed is equal to 1, and the feature values ​​included in the first feature maps correspond to the same target gain value, i.e., the first feature map corresponds to one target gain value. In this case, the feature values ​​included in the first feature maps are multiplied by the corresponding target gain value to change the distribution of the N first feature values ​​included in the multiple first feature maps. A larger target compression bit rate indicates a more dispersed distribution of the N first feature values.

[0154] In this embodiment of the present application, after feature extraction is performed on the first image to obtain multiple first feature maps, some of the extracted first feature maps need to be processed (gain processing may not be performed on the first feature maps corresponding to some semantic channels that have a relatively small impact on the compression result). The number of extracted first feature maps that need to be processed is equal to 1, and the feature values ​​included in the first feature maps correspond to different target gain values, i.e., the first feature maps correspond to multiple target gain values ​​(different feature values ​​in the same feature map may correspond to different target gain values). In this case, the feature values ​​included in the first feature maps are multiplied by the corresponding target gain values ​​to change the distribution of the N first feature values ​​included in the multiple first feature maps. A larger target compression bit rate indicates a more dispersed distribution of the N first feature values.

[0155] It should be noted that the gain processing may be performed only on the first feature values ​​included in the first feature map.

[0156] It should be noted that when the same scale gain processing is performed on the feature values ​​of the semantic channels, i.e., the first feature values ​​included in the multiple first feature maps corresponding to all the semantic channels correspond to the same target gain value, the information entropy of the N first feature values ​​may be changed, but the compression effect is relatively low. Therefore, the basic gain calculation unit is set at the semantic channel level (the first feature values ​​included in each of the first feature maps corresponding to at least two of all the semantic channels correspond to different target gain values) or the feature value level (the first feature values ​​included in each of the first feature maps corresponding to the semantic channels correspond to different target gain values), so that a relatively good compression effect can be achieved.

[0157] In the following, it will be described how to obtain M target gain values ​​that can implement the aforementioned technical effects.

[0158] 1.Manual decision method In this embodiment of the present application, the objective function mapping relationship may be determined manually. If the first feature value included in the first feature map corresponding to each semantic channel corresponds to the same target gain value, the input of the objective function mapping relationship may be the semantic channel and the target compression bit rate, and the output of the objective function mapping relationship is the corresponding target gain value (since the first feature value included in the first feature map corresponds to the same target gain value, all the target gain values ​​corresponding to the semantic channels can be represented by using one target gain value). For example, the target gain value corresponding to each semantic channel may be determined by using a linear function, a quadratic function, a cubic function, or a quartic function. FIG. 6 illustrates an objective function mapping relationship according to an embodiment of the present application. As shown in FIG. 6, the objective function mapping relationship is a linear function, the input of the function is the semantic channel sequence number (e.g., there are semantic channel sequence numbers 1 to 192), and the output of the function is the target mapping function, and each target compression bit rate corresponds to a different objective function mapping relationship. A larger target compression bit rate corresponds to a smaller slope of the objective function mapping relationship. The approximate distribution law of a quadratic or cubic nonlinear function is similar, and the details will not be described here.

[0159] In this embodiment of the present application, the target gain value corresponding to each of the M first feature values ​​can be manually determined. The specific setting manner is not limited in the present application, provided that the larger the target compression bit rate is, the more dispersed the distribution of the N first feature values ​​is.

[0160] 2. Training method In this embodiment of the present application, obtaining M target gain values ​​corresponding to each target compression bit rate in a training manner needs to be combined with the processing on the decoding side. Therefore, obtaining M target gain values ​​corresponding to each target compression bit rate in a training manner will be described in detail in the subsequent embodiment, and the details will not be described here.

[0161] 305. Quantize and entropy code the at least one processed first feature map to obtain encoded data, where the at least one processed first feature map includes M second feature values.

[0162] In this embodiment of the present application, the corresponding first feature values ​​are respectively processed based on M target gain values ​​to obtain M second feature values, and then quantization and entropy coding may be performed on the at least one processed first feature map to obtain encoded data. The at least one processed first feature map includes the M second feature values.

[0163] In this embodiment of the present application, the N first feature values ​​are converted into quantization centers according to a specified rule to facilitate subsequent entropy coding. The quantization operation can convert the N first feature values ​​from floating-point numbers into a bitstream (e.g., a bitstream using a specific bit integer, such as an 8-bit integer or a 4-bit integer). In some embodiments, the quantization operation may be performed on the N first feature values ​​by performing, but is not limited to, rounding.

[0164] In this embodiment of the present application, the information entropy of the quantized data obtained by quantizing at least one processed first feature map satisfies a preset condition, and the preset condition is related to a target compression bit rate. Specifically, a larger target compression bit rate indicates a larger information entropy of the quantized data.

[0165] In this embodiment of the present application, the probability estimates of the points in the output features may be obtained by using an entropy estimation network, and the output features are entropy coded by using the probability estimates to obtain a binary bit stream. It should be noted that the entropy coding process in this application may use existing entropy coding techniques, and the details will not be described in this application.

[0166] In this embodiment of the present application, the difference between the compression bit rate corresponding to the encoded data and the target compression bit rate falls within a preset range. The preset range may be selected in practical application. As long as the difference between the compression bit rate corresponding to the encoded data and the target compression bit rate falls within an acceptable range, the specific preset range is not limited in the present application.

[0167] In this embodiment of the present application, after the encoded data is obtained, the encoded data may be sent to a terminal device for decompression. In this case, the image processing device for decompression can decompress the data. Alternatively, the terminal device for compression can store the encoded data in a storage device. When the encoded data is needed, the terminal device can obtain the encoded data from the storage device and decompress the encoded data.

[0168] Optionally, in one embodiment, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponds to M first gain values, the second compression bit rate corresponds to M second gain values, and the M target gain values ​​are obtained by performing an interpolation operation on the M first gain values ​​and the M second gain values. In this embodiment of the present application, the M first values ​​include a first target gain value, the M second gain values ​​include a second target gain value, and the M target gain values ​​include a third target gain value, and the first target gain value, the second target gain value, and the third target gain value correspond to a same feature value of the M first feature values, and the third target gain value is obtained by performing an interpolation operation on the first target gain value and the second target gain value.

[0169] In this embodiment of the present application, the compression effects of multiple compression bit rates may be implemented in a single model. Specifically, to implement the compression effects for different compression bit rates, different target gain values ​​may be correspondingly set for multiple target compression bit rates. Then, an interpolation operation may be performed on the target gain values ​​by using an interpolation algorithm to obtain a new gain value of any compression effect within the compression bit rate range. Specifically, the M first gain values ​​include a first target gain value, the M second gain values ​​include a second target gain value, and the M target gain values ​​include a third target gain value, where the first target gain value, the second target gain value and the third target gain value correspond to the same feature value of the M first feature values, and the third target gain value is obtained by performing an interpolation operation on the first target gain value and the second target gain value. The interpolation operation may be performed based on the following formula: m l =[(m i ) l (m j ) 1-l ], where m l represents the third target gain value, and m i represents the first target gain value, mj represents the second target gain value, and m l , m i , and m j correspond to the same feature value, and l ∈ (0,1) is an adjustment factor, which may be determined based on the size of the target compressed bitrate.

[0170] In this embodiment of the present application, after M target gain values ​​corresponding to each of a plurality of compression bit rates are obtained, when compression corresponding to the target compression bit rate is performed, two groups of target gain values ​​(each group includes M target gain values) corresponding to two compression bit rates adjacent to the target compression bit rate may be determined from the plurality of compression bit rates, and the above-mentioned interpolation process is performed on the two groups of target gain values ​​to obtain M target gain values ​​corresponding to the target compression bit rate. In this embodiment of the present application, any compression effect of the AI ​​compression model in the compression bit rate interval may be implemented.

[0171] In this embodiment of the present application, each of the M target gain values ​​corresponds to one inverse gain value, and the inverse gain value is used to process the feature value obtained in the decoding process of the encoded data, and the product of each of the M target gain values ​​and the corresponding inverse gain value falls within a preset range. The inverse gain process on the decoding side will be described in the subsequent embodiment, and the details will not be described here.

[0172] This embodiment of the present application provides an image processing method. A first image is obtained, and feature extraction is performed on the first image to obtain at least one first feature map, the at least one first feature map includes N first feature values, N is a positive integer, a target compression bit rate is obtained, the target compression bit rate corresponds to M target gain values, each target gain value corresponds to one first feature value, M is a positive integer less than or equal to N, the corresponding first feature value is respectively processed based on the M target gain values ​​to obtain M second feature values, and quantization and entropy coding are performed on the at least one processed first feature map to obtain encoded data, the at least one processed first feature map includes M second feature values. In the above scheme, different target gain values ​​are set for different target compression bit rates to implement compression bit rate control.

[0173] 7 shows an embodiment of an image processing method according to an embodiment of the present application. As shown in FIG. 7, the image processing method provided in this embodiment includes the following steps:

[0174] 701. Obtain encoded data.

[0175] In this embodiment of the present application, the encoded data obtained in FIG. 3 and corresponding embodiments may be obtained.

[0176] In this embodiment of the present application, after the encoded data is obtained, the encoded data may be sent to a terminal device for decompression. In this case, the image processing device for decompression can obtain and decompress the encoded data. Alternatively, the terminal device for compression can store the encoded data in a storage device. When the encoded data is needed, the terminal device can obtain the encoded data from the storage device and decompress the encoded data.

[0177] 702. Perform entropy decoding on the encoded data to obtain at least one second feature map, the at least one second feature map including N third feature values, where N is a positive integer.

[0178] In this embodiment of the present application, the encoded data may be decoded by using an entropy decoding technique in the prior art to obtain a reconstructed output feature (at least one second feature map), which includes N third feature values.

[0179] It should be noted that the at least one second feature map in this embodiment of the present application may be the same as the at least one processed first feature map mentioned above.

[0180] 703. Obtain M target inverse gain values, each target inverse gain value corresponding to one third feature value, where M is a positive integer less than or equal to N.

[0181] Optionally, in one embodiment, a target compression bitrate may be obtained, and M target inverse gain values ​​corresponding to the target compression bitrate may be determined based on a target mapping relationship. The target mapping relationship is used to indicate a correlation between the compression bitrate and the inverse gain vector. The target mapping relationship includes a plurality of compression bitrates, a plurality of inverse gain vectors, and a correlation between the plurality of compression bitrates and the plurality of inverse gain vectors, the target compression bitrate is one of the plurality of compression bitrates, and the M target inverse gain values ​​are one element of the plurality of inverse gain vectors, or the target mapping relationship includes a target function mapping relationship, and when an input of the target function relationship includes the target compression bitrate, an output of the target function relationship includes the M target inverse gain values.

[0182] In this embodiment of the present application, the target inverse gain value may be obtained in the step of obtaining the target gain value in the embodiment corresponding to Fig. 3. This is not limited here.

[0183] Optionally, in one embodiment, the at least one second feature map includes a second target feature map, the second target feature map including P third feature values, all of the P third feature values ​​corresponding to the same target inverse gain value, where P is a positive integer less than or equal to M.

[0184] Optionally, in one embodiment, the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0185] 704. Process the corresponding third feature values ​​based on the M target inverse gain values, respectively, to obtain M fourth feature values.

[0186] In this embodiment of the present application, the M fourth feature values ​​may be obtained by performing multiplication operations on the M target inverse gain values ​​and the corresponding third feature values ​​individually. Specifically, in this embodiment of the present application, the M third feature values ​​in at least one second feature map are multiplied by the corresponding inverse gain values ​​respectively to obtain the M fourth feature values, so that the at least one second feature map obtained after the inverse gain processing contains the M fourth feature values. The inverse gain processing is combined with the gain processing in the embodiment corresponding to FIG. 3, so that normal image analysis can be guaranteed.

[0187] 705. Perform image reconstruction on the at least one processed second feature map to obtain a second image, where the at least one processed second feature map includes the M fourth feature values.

[0188] In this embodiment of the present application, after the M fourth feature values ​​are obtained, image reconstruction may be performed on the at least one processed second feature map to obtain a second image. The at least one processed second feature map includes the M fourth feature values. The at least one second feature map is analyzed in the manner described above and reconstructed into a second image.

[0189] Optionally, in one embodiment, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponds to M first inverse gain values, the second compression bit rate corresponds to M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values. In this embodiment of the present application, the M first inverse gain values ​​include a first target inverse gain value, the M second inverse gain values ​​include a second target inverse gain value, and the M target inverse gain values ​​include a third target inverse gain value, and the first target inverse gain value, the second target inverse gain value, and the third target inverse gain value correspond to the same feature value of the M first feature values, and the third target inverse gain value is obtained by performing an interpolation operation on the first target inverse gain value and the second target inverse gain value.

[0190] In this embodiment of the present application, each of the M target gain values ​​corresponds to one inverse gain value, and the inverse gain value is used to process the feature value obtained in the decoding process of the encoded data, and the product of each of the M target gain values ​​and the corresponding inverse gain value falls within a preset range, that is, there is a specific value relationship between the target gain value and the inverse gain value corresponding to the same feature value: The product of the two values ​​falls within a preset range. The preset range may be a value range close to the value "1", and is not limited here.

[0191] This embodiment of the present application provides an image processing method. Encoded data is obtained, and entropy decoding is performed on the encoded data to obtain at least one second feature map, the at least one second feature map includes N third feature values, N is a positive integer, and M target inverse gain values ​​are obtained, each target inverse gain value corresponds to one third feature value, and M is a positive integer less than or equal to N. The corresponding third feature values ​​are respectively processed based on the M target inverse gain values ​​to obtain M fourth feature values, and image reconstruction is performed on the at least one processed second feature map to obtain a second image, the at least one processed second feature map includes M fourth feature values. In the above scheme, different target inverse gain values ​​are set for different target compression bit rates to implement compression bit rate control.

[0192] Next, the architecture of a variational autoencoder (VAE) is used as an example to explain the image compression method provided in the embodiments of this application. A variational autoencoder is an autoencoder used for data compression or noise reduction.

[0193] FIG. 8 illustrates an image compression procedure according to an embodiment of the present application.

[0194] This embodiment provides an explanation by using an example in which the target gain values ​​corresponding to the same semantic channel are the same, and the target inverse gain values ​​corresponding to the same semantic channel are the same. There are 192 semantic channels, and training needs to be done at four specified code points (four compression bit rates) during training. Each compression bit rate corresponds to one target gain vector and one target inverse gain vector. The target gain vector m i is a vector of size 192 × 1, corresponding to the compression bit rate. The target inverse gain vector

number

number

[0195] 801. Obtain output features y after a first image enters an encoding network.

[0196] 802. Output features obtained after gain processing

number

[0197] 803. Features

number

number

[0198] 804. Obtain probability estimates of points in the output features by using an entropy estimation module, and perform entropy coding on the output features by using the probability estimates to obtain a binary bit stream.

[0199] 805. Reconstructed output features

number

[0200] 806. To obtain the output feature y' obtained after the inverse gain processing, the output feature

number

number

[0201] 807. After the output features enter the decoding network, the output features y′ are analyzed and reconstructed into a second image.

[0202] Please refer to Figure 9. The left diagram of Figure 9 shows a comparison between the rate-distortion performance (non-dashed line) of the single model of the present embodiment and the rate-distortion performance (dashed line) of the four compression models separately trained by using the VAE method in the prior art under the condition that the multi-scale structural similarity index measure (MS-SSIM) is used as the evaluation index, where the abscissa is BPP and the ordinate is MS-SSIM. The right diagram of Figure 9 shows a comparison between the rate-distortion performance (non-dashed line) of the single model of the present embodiment and the rate-distortion performance (dashed line) of the four compression models separately trained by using the VAE method in the prior art under the condition that the peak signal to noise ratio (PSNR) is used as the evaluation index, where the abscissa is BPP and the ordinate is PSNR. In this embodiment, on the premise that the number of model parameters is basically consistent with the number of model parameters of a single model in the VAE method, it can be seen that the compression effect of any bit rate can be implemented based on both evaluation indexes, the compression effect is not worse than the multi-model implementation effect of the VAE method, and the memory amount of the model can be reduced by N times (N is the number of models required in the VAE method to implement the compression effects of different bit rates in this embodiment of the present invention).

[0203] 10 shows the training process according to one embodiment of the present application. As shown in FIG. 10, the loss function of the model in this embodiment is as follows: loss=l d +β l r , where l d is the distortion loss of the second image relative to the first image, calculated based on the evaluation index, and l r is the bitrate loss (or called the bitrate estimate) obtained by the entropy estimation network through calculation, and β is a Lagrangian coefficient for adjusting the tradeoff between the distortion loss and the bitrate estimate.

[0204] To obtain the gain matrix and inverse gain matrix {M,M'} corresponding to different compression bit rates, the model training process may be shown in Figure 10. The Lagrangian coefficient β in the loss function is constantly transformed in the model training process to obtain the corresponding gain and inverse gain vectors

number

[0205] For example, the compression effect of four compression bit rates can be implemented in a single model. The four gain vectors obtained by training are multiplied by the corresponding inverse gain vector. The multiplication results of the corresponding elements in the target gain vector and the target inverse gain vector corresponding to different compression bit rates are approximately equal, so that the following relationship can be obtained:

number

number

number

[0206] To implement continuous bitrate adjustment in a single model, in this embodiment, the following derivation may be performed by using the above formula:

number

[0207] In this embodiment of the present application, an interpolation operation may be performed on four adjacent gain and inverse gain vector pairs obtained through training to obtain a new gain and inverse gain vector pair.

[0208] In order to obtain the gain matrix M matching different compression bit rates, the training process is as follows: In this embodiment, the Lagrangian coefficients in the loss function are constantly transformed in the model training process, and the corresponding gain vector m i and the inverse gain vector

number

number

[0209] In this embodiment of the present application, the gain vector m i and the inverse gain vector

number

[0210] In this embodiment, on the premise that the number of model parameters is basically consistent with the number of model parameters of a single VAE method model, the compression effect of any bit rate can be implemented, the compression effect is not worse than the effect of independent training at each bit rate, and the model storage amount can be reduced by N times (N is the number of models required to implement the compression effect of different bit rates of this embodiment of the present invention in the VAE method).

[0211] Please note that only VAE is used above as an architecture for explanation. In practical application, the image compression method may be further applied to another AI compression model architecture (such as an auto-encoder or another image compression model). This is not limited in this application.

[0212] 12 is a diagram of a system architecture of an image processing system according to an embodiment of the present application. In FIG. 12, the image processing system 200 includes an execution device 210, a training device 220, a database 230, a client device 240, and a data storage system 250. The execution device 210 includes a computation module 211.

[0213] The database 230 stores a set of first images. The training device 220 generates a target model / rule 201 used to process the first images, and performs iterative training on the target model / rule 201 by using the first images in the database to obtain a mature target model / rule 201. This embodiment of the present application provides an explanation by using an example in which the target model / rule 201 includes a second encoding / decoding network, M target gain values ​​and M target inverse gain values ​​corresponding to each compression bit rate.

[0214] The second encoding / decoding network and the M target gain values ​​and M target inverse gain values ​​corresponding to each compression bit rate obtained by the training device 220 may be applied to different systems or devices, such as a mobile phone, a tablet computer, a notebook computer, a VR device, or a surveillance system. The execution device 210 may call or store data, instructions, etc. in the data storage system 250. The data storage system 250 may be located in the execution device 210 or the data storage system 250 may be an external memory to the execution device 210.

[0215] The computation module 211 performs feature extraction on the first image received by the client device 240 by using the second encoding / decoding network to obtain at least one first feature map, the at least one first feature map including N first feature values, N being a positive integer, obtain a target compression bit rate, the target compression bit rate corresponding to M target gain values, each target gain value corresponding to one first feature value, M being a positive integer less than or equal to N, respectively processing the corresponding first feature values ​​based on the M target gain values ​​to obtain M second feature values, and performing quantization and entropy coding on the at least one processed first feature map to obtain encoded data, the at least one processed first feature map including the M second feature values.

[0216] The computation module 211 further performs entropy decoding on the encoded data by using a second encoding / decoding network to obtain at least one second feature map, where the at least one second feature map includes N third feature values, where N is a positive integer, obtain M target inverse gain values, where each target inverse gain value corresponds to one third feature value, where M is a positive integer less than or equal to N, respectively process the corresponding third feature values ​​based on the M target inverse gain values ​​to obtain M fourth feature values, and perform image reconstruction on the at least one processed second feature map to obtain a second image, where the at least one processed second feature map includes the M fourth feature values.

[0217] In some embodiments of the present application, see Fig. 12. The execution device 210 and the client device 240 may be independent devices. The I / O interface 212 is configured in the execution device 210 to exchange data with the client device 240. A "user" can input a first image to the I / O interface 212 by using the client device 240, and the execution device 210 returns the second image to the client device 240 by using the I / O interface 212 to provide the second image to the user.

[0218] It should be noted that FIG. 12 is merely a schematic diagram of the architecture of an image processing system according to an embodiment of the present invention, and the location relationship of devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in some other embodiments of the present application, the execution device 210 may be configured in the client device 240. For example, if the client device is a mobile phone or tablet computer, the execution device 210 may be a module configured to process array images in a host central processing unit (Host CPU) of the mobile phone or tablet computer, or the execution device 210 may be a graphics processing unit (GPU) or a neural network processing unit (NPU) in the mobile phone or tablet computer. The GPU or NPU is installed in the host central processing unit as a coprocessor, and the host central processing unit assigns tasks to the GPU or NPU.

[0219] With reference to the foregoing description, the following begins with describing the specific implementation procedure of the training phase of the image processing method provided in the embodiment of the present application.

[0220] 1. Training Phase For details, please refer to Figure 13. Figure 13 is a schematic flow chart of an image processing method according to an embodiment of the present application. The image processing method provided in this embodiment of the present application may include the following steps.

[0221] 1301. A first image is acquired.

[0222] 1302. Perform feature extraction on the first image based on the encoding network to obtain at least one first feature map, where the at least one first feature map includes N first feature values, where N is a positive integer.

[0223] 1303. Obtain a target compression bit rate, where the target compression bit rate corresponds to M initial gain values ​​and M initial inverse gain values, each initial gain value corresponds to one first feature value, each initial inverse gain value corresponds to one third feature value, and M is a positive integer less than or equal to N.

[0224] 1304. Process the corresponding first feature values ​​based on the M initial gain values, respectively, to obtain M second feature values.

[0225] 1305. To obtain encoded data and bit rate loss, perform quantization and entropy coding on the at least one processed first feature map based on a quantization network and an entropy coding network, and the at least one first feature map obtained after gain processing includes M second feature values.

[0226] 1306. Perform entropy decoding on the encoded data based on the entropy decoding network to obtain at least one second feature map, where the at least one second feature map includes M third feature values, and each third feature value corresponds to one first feature value.

[0227] 1307. Process the corresponding third feature values ​​based on the M initial inverse gain values, respectively, to obtain M fourth feature values.

[0228] 1308. To obtain a second image, perform image reconstruction on the at least one processed second feature map based on the decoding network, where the at least one processed feature map includes M fourth feature values.

[0229] 1309. Obtain a distortion loss of the second image relative to the first image.

[0230] 1310. Perform joint training on a first encoding / decoding network, M initial gain values, and M initial inverse gain values ​​by using a loss function until an image distortion value between a first image and a second image reaches a first preset degree, wherein the image distortion value is related to a bitrate loss and a distortion loss, and the encoding / decoding network includes a coding network, a quantization network, an entropy coding network, and an entropy decoding network.

[0231] 1311. Output a second encoding / decoding network, M target gain values, and M target inverse gain values, where the second encoding / decoding network is a model obtained after iterative training is performed on the first encoding / decoding network, and the M target gain values ​​and the M target inverse gain values ​​are obtained after iterative training is performed on the M initial gain values ​​and the M initial inverse gain values.

[0232] Please refer to the description in the above embodiment for a specific description of steps 1301 to 1311. This is not limited here.

[0233] Optionally, an information entropy of the quantized data obtained by quantizing the at least one processed first feature map satisfies a preset condition, the preset condition being related to a target compression bitrate.

[0234] Optionally, the pre-defined conditions include at least: This includes indicating that the higher the target compression bit rate, the greater the information entropy of the quantized data.

[0235] Optionally, the M second feature values ​​are obtained by performing multiplication operations on the M initial gain values ​​and the corresponding first feature values ​​respectively.

[0236] Optionally, the M fourth feature values ​​are obtained by performing multiplication operations on the M initial inverse gain values ​​and the corresponding third feature values ​​respectively.

[0237] Optionally, the product of each of the M target gain values ​​and the corresponding target inverse gain value falls within a preset range, and the product of each of the M initial gain values ​​and the corresponding initial inverse gain value falls within a preset range.

[0238] According to the embodiment corresponding to Fig. 1-Fig. 13, in order to better implement the aforementioned solution in the embodiment of the present application, the following further provides a related device configured to implement the aforementioned solution. Please refer to Fig. 14 for details. Fig. 14 is a schematic diagram of the configuration of an image processing device 1400 according to an embodiment of the present application. The image processing device 1400 may be a terminal device or a server, and the image processing device 1400 includes: an acquisition module 1401 configured to acquire a first image; a feature extraction module 1402 configured to perform feature extraction on the first image to obtain at least one first feature map, the at least one first feature map comprising N first feature values, where N is a positive integer; The obtaining module 1401 is further configured to obtain a target compression bit rate, where the target compression bit rate corresponds to M target gain values, each target gain value corresponds to one first feature value, where M is a positive integer less than or equal to N; a gain module 1403 configured to process each of the corresponding first feature values ​​based on the M target gain values ​​to obtain M second feature values; a quantization and entropy coding module 1404 configured to perform quantization and entropy coding on the at least one processed first feature map to obtain coded data, the at least one processed first feature map including M second feature values; Equipped with.

[0239] Optionally, an information entropy of the quantized data obtained by quantizing the at least one processed first feature map satisfies a preset condition, the preset condition being related to a target compression bitrate.

[0240] Optionally, the pre-defined conditions include at least: This includes indicating that the higher the target compression bit rate, the greater the information entropy of the quantized data.

[0241] Optionally, the difference between the compression bit rate corresponding to the encoded data and the target compression bit rate falls within a preset range.

[0242] Optionally, the M second feature values ​​are obtained by performing multiplication operations on the M target gain values ​​and the corresponding first feature values ​​respectively.

[0243] Optionally, the at least one first feature map includes a first target feature map, the first target feature map including P first feature values, all of the P first feature values ​​corresponding to the same target gain value, where P is a positive integer less than or equal to M.

[0244] Optionally, the apparatus further comprises: a determination module configured to determine M target gain values ​​corresponding to a target compression bit rate based on a target mapping relationship, the target mapping relationship being used to indicate a correlation between the compression bit rate and the M target gain values; the target mapping relationship includes a plurality of compressed bit rates, a plurality of gain vectors, and a correlation between the plurality of compressed bit rates and the plurality of gain vectors, the target compressed bit rate being one of the plurality of compressed bit rates, and the M target gain values ​​being one element of the plurality of gain vectors; or If the target mapping relationship comprises an objective function mapping relationship, and the input of the objective function relationship comprises the target compression bit rate, then the output of the objective function relationship comprises the M target gain values.

[0245] Optionally, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponding to the M first gain values ​​and the second compression bit rate corresponding to the M second gain values, and the M target gain values ​​are obtained by performing an interpolation operation on the M first gain values ​​and the M second gain values.

[0246] Optionally, the M first gain values ​​include a first target gain value, the M second gain values ​​include a second target gain value, and the M target gain values ​​include a third target gain value, wherein the first target gain value, the second target gain value, and the third target gain value correspond to a same feature value among the M first feature values, and the third target gain value is obtained by performing an interpolation operation on the first target gain value and the second target gain value.

[0247] Optionally, the first image includes a target object, and the M first feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0248] Optionally, each of the M target gain values ​​corresponds to an inverse gain value, and the inverse gain value is used to process the feature values ​​obtained in the decoding process of the encoded data, and a product of each of the M target gain values ​​and the corresponding inverse gain value falls within a preset range.

[0249] Optionally, the apparatus further comprises: a decoding module configured to perform entropy decoding on the encoded data to obtain at least one second feature map, the at least one second feature map including N third feature values, each third feature value corresponding to one first feature value; The obtaining module is further configured to obtain M target inverse gain values, each target inverse gain value corresponding to one third feature value; This device is an inverse gain module configured to perform gain processing on corresponding third feature values ​​based on the M target inverse gain values, respectively, to obtain M fourth feature values; a reconstruction module configured to perform image reconstruction on at least one second feature map obtained after the inverse gain processing to obtain a second image, wherein the at least one second feature map obtained after the inverse gain processing includes M fourth feature values; It further comprises:

[0250] Optionally, the M fourth feature values ​​are obtained by performing multiplication operations on the M target inverse gain values ​​and the corresponding third feature values ​​respectively.

[0251] Optionally, the at least one second feature map includes a second target feature map, the second target feature map including P third feature values, all of the P third feature values ​​corresponding to the same target inverse gain value, where P is a positive integer less than or equal to M.

[0252] Optionally, the decision module: The method is further configured to determine M target inverse gain values ​​corresponding to the target compression bitrates based on the target mapping relationship, the target mapping relationship being used to indicate a correlation between the compression bitrates and the inverse gain vectors.

[0253] Optionally, the target mapping relationship includes a plurality of compressed bit rates, a plurality of inverse gain vectors, and a correlation relationship between the plurality of compressed bit rates and the plurality of inverse gain vectors, where the target compressed bit rate is one of the plurality of compressed bit rates, and the M target inverse gain values ​​are elements of one of the plurality of inverse gain vectors.

[0254] Optionally, the target mapping relationship comprises an objective function mapping relationship, where an input of the objective function relationship comprises the target compression bitrate and an output of the objective function relationship comprises the M target inverse gain values.

[0255] Optionally, the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0256] Optionally, the product of each of the M target gain values ​​and the corresponding target inverse gain value falls within a preset range.

[0257] Optionally, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponds to the M first inverse gain values, and the second compression bit rate corresponds to the M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values.

[0258] Optionally, the M first inverse gain values ​​include a first target inverse gain value, the M second inverse gain values ​​include a second target inverse gain value, and the M target inverse gain values ​​include a third target inverse gain value, wherein the first target inverse gain value, the second target inverse gain value, and the third target inverse gain value correspond to a same feature value among the M first feature values, and the third target inverse gain value is obtained by performing an interpolation operation on the first target inverse gain value and the second target inverse gain value.

[0259] This embodiment of the present application provides an image processing apparatus 1400. The acquisition module 1401 acquires a first image. The feature extraction module 1402 performs feature extraction on the first image to obtain at least one first feature map, where the at least one first feature map includes N first feature values, where N is a positive integer. The acquisition module 1401 acquires a target compression bit rate, where the target compression bit rate corresponds to M target gain values, where each target gain value corresponds to one first feature value, where M is a positive integer less than or equal to N. The gain module 1403 processes the corresponding first feature values ​​according to the M target gain values ​​to obtain M second feature values. The quantization and entropy coding module 1404 performs quantization and entropy coding on the at least one processed first feature map to obtain encoded data, where the at least one processed first feature map includes M second feature values. In the aforementioned scheme, different target gain values ​​are set for different target compression bit rates to implement compression bit rate control.

[0260] 15 is a schematic diagram of a configuration of an image processing device 1500 according to an embodiment of the present invention. The image processing device 1500 may be a terminal device or a server. an acquisition module 1501 configured to acquire encoded data; a decoding module 1502 configured to perform entropy decoding on the encoded data to obtain at least one second feature map, the at least one second feature map including N third feature values, where N is a positive integer; The obtaining module 1501 is further configured to obtain M target inverse gain values, each target inverse gain value corresponding to one third feature value, where M is a positive integer equal to or less than N; an inverse gain module 1503 configured to process each of the corresponding third feature values ​​based on the M target inverse gain values ​​to obtain M fourth feature values; a reconstruction module 1504 configured to perform image reconstruction on the at least one processed second feature map to obtain a second image, wherein the at least one processed second feature map includes the M fourth feature values; Equipped with.

[0261] Optionally, the M fourth feature values ​​are obtained by performing multiplication operations on the M target inverse gain values ​​and the corresponding third feature values ​​respectively.

[0262] Optionally, the at least one second feature map includes a second target feature map, the second target feature map including P third feature values, all of the P third feature values ​​corresponding to the same target inverse gain value, where P is a positive integer less than or equal to M.

[0263] Optionally, the obtaining module is further configured to obtain a target compression bitrate; This device is A determining module determines M target inverse gain values ​​corresponding to a target compression bit rate based on a target mapping relationship, the target mapping relationship being used to indicate a correlation between the compression bit rate and the inverse gain vector; the target mapping relationship includes a plurality of compression bit rates, a plurality of inverse gain vectors, and a correlation between the plurality of compression bit rates and the plurality of inverse gain vectors, the target compression bit rate being one of the plurality of compression bit rates, and the M target inverse gain values ​​being one element of the plurality of inverse gain vectors; or If the target mapping relationship comprises an objective function mapping relationship, and the input of the objective function relationship comprises the target compression bit rate, then the output of the objective function relationship comprises the M target inverse gain values.

[0264] Optionally, the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0265] Optionally, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponds to the M first inverse gain values, and the second compression bit rate corresponds to the M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values.

[0266] Optionally, the M first inverse gain values ​​include a first target inverse gain value, the M second inverse gain values ​​include a second target inverse gain value, and the M target inverse gain values ​​include a third target inverse gain value, wherein the first target inverse gain value, the second target inverse gain value, and the third target inverse gain value correspond to a same feature value among the M first feature values, and the third target inverse gain value is obtained by performing an interpolation operation on the first target inverse gain value and the second target inverse gain value.

[0267] This embodiment of the present invention provides an image processing apparatus. The acquisition module 1501 acquires encoded data. The decoding module 1502 performs entropy decoding on the encoded data to obtain at least one second feature map, where the at least one second feature map includes N third feature values, where N is a positive integer. The acquisition module 1501 acquires M target inverse gain values, where each target inverse gain value corresponds to one third feature value, where M is a positive integer equal to or less than N. The inverse gain module 1503 processes the corresponding third feature values ​​according to the M target inverse gain values, respectively, to obtain M fourth feature values. The reconstruction module 1504 performs image reconstruction on the at least one processed second feature map to obtain a second image, where the at least one processed second feature map includes M fourth feature values. In the above scheme, different target gain values ​​are set for different target compression bit rates to implement compression bit rate control.

[0268] 16 is a schematic diagram of the configuration of an image processing device 1600 according to an embodiment of the present application. The image processing device 1600 may be a terminal device or a server. The image processing device 1600 includes: an acquisition module 1601 configured to acquire a first image; a feature extraction module 1602 configured to perform feature extraction on the first image based on the encoding network to obtain at least one first feature map, the at least one first feature map including N first feature values, where N is a positive integer; The obtaining module 1601 is further configured to obtain a target compression bit rate, the target compression bit rate corresponding to M initial gain values ​​and M initial inverse gain values, each initial gain value corresponding to one first feature value, each initial inverse gain value corresponding to one third feature value, and M is a positive integer less than or equal to N; a gain module 1603 configured to process each of the corresponding first feature values ​​based on the M initial gain values ​​to obtain M second feature values; a quantization and entropy coding module 1604 configured to perform quantization and entropy coding on the at least one processed first feature map based on a quantization network and an entropy coding network to obtain coded data and a bit rate loss, where the at least one first feature map obtained after the gain processing includes M second feature values; a decoding module 1605 configured to perform entropy decoding on the encoded data based on the entropy decoding network to obtain at least one second feature map, the at least one second feature map including M third feature values, each third feature value corresponding to one first feature value; an inverse gain module 1606 configured to process corresponding third feature values, respectively, based on the M initial inverse gain values ​​to obtain M fourth feature values; a reconstruction module 1607 configured to perform image reconstruction on the at least one processed second feature map based on the decoding network to obtain a second image, wherein the at least one processed feature map includes M fourth feature values; the acquisition module 1601 is further configured to acquire a distortion loss of the second image relative to the first image; a training module 1608 configured to perform joint training on a first encoding / decoding network, the M initial gain values, and the M initial inverse gain values ​​by using a loss function until an image distortion value between the first image and the second image reaches a first preset degree, the image distortion value being related to a bitrate loss and a distortion loss, and the encoding / decoding network includes an encoding network, a quantization network, an entropy encoding network, and an entropy decoding network; an output module 1609 configured to output a second encoding / decoding network, M target gain values, and M target inverse gain values, where the second encoding / decoding network is a model obtained after iterative training is performed on the first encoding / decoding network, and the M target gain values ​​and the M target inverse gain values ​​are obtained after iterative training is performed on the M initial gain values ​​and the M initial inverse gain values; Includes.

[0269] Optionally, an information entropy of the quantized data obtained after the at least one first feature map obtained after the gain processing is quantized satisfies a preset condition, the preset condition being related to a target compression bit rate, and N is a positive integer greater than or equal to M.

[0270] Optionally, the pre-defined conditions include at least: This includes indicating that the higher the target compression bit rate, the greater the information entropy of the quantized data.

[0271] Optionally, the M second feature values ​​are obtained by performing multiplication operations on the M target gain values ​​and the corresponding first feature values ​​respectively.

[0272] Optionally, the at least one first feature map includes a first target feature map, the first target feature map including P first feature values, all of the P first feature values ​​corresponding to the same target gain value, where P is a positive integer less than or equal to M.

[0273] Optionally, the first image includes a target object, and the M first feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0274] Optionally, the product of each of the M target gain values ​​and the corresponding target inverse gain value falls within a preset range, and the product of each of the M initial gain values ​​and the corresponding initial inverse gain value falls within a preset range.

[0275] The following describes an execution device provided in an embodiment of the present application. FIG. 17 is a schematic diagram of the structure of an execution device according to an embodiment of the present application. The execution device 1700 may be specifically represented as a virtual reality VR device, a mobile phone, a tablet computer, a notebook computer, an intelligent wearable device, a monitoring data processing device, etc. This is not limited here. The image processing device described in the embodiment corresponding to FIG. 14 and FIG. 15 may be deployed in the execution device 1700 to implement the functions of the image processing device of the embodiment corresponding to FIG. 14 and FIG. 15. Specifically, the execution device 1700 includes a receiver 1701, a transmitter 1702, a processor 1703, and a memory 1704 (there may be one or more processors 1703 in the execution device 1700, and one processor is used as an example in FIG. 17). The processor 1703 may include an application processor 17031 and a communication processor 17032. In some embodiments of the present application, the receiver 1701, the transmitter 1702, the processor 1703, and the memory 1704 may be connected using a bus or in another manner.

[0276] The memory 1704 may include read-only memory and random access memory to provide instructions and data to the processor 1703. A portion of the memory 1704 may further include non-volatile random access memory (NVRAM). The memory 1704 stores processor-executable operating instructions, executable modules, data structures, a subset thereof, or an extended set thereof. The operating instructions may include various operating instructions for performing various operations.

[0277] The processor 1703 controls the operation of the execution device. During a particular application, the components of the execution device are coupled to each other by using a bus system. In addition to a data bus, the bus system may further include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, the various types of buses in the figures are labeled as a bus system.

[0278] The methods disclosed in the above-mentioned embodiments of the present application may be applied to or implemented by the processor 1703. The processor 1703 may be an integrated circuit chip and has signal processing capabilities. In the implementation process, the steps in the above-mentioned methods may be implemented by using hardware integrated logic circuits in the processor 1703 or by using instructions in the form of software. The processor 1703 may be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, or may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or another programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor 1703 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the method disclosed with reference to the embodiments of the present application may be directly performed and completed by a hardware decoding processor, or may be performed and completed by using a combination of hardware modules and software modules in the decoding processor. The software modules may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in the memory 1704, and the processor 1703 reads the information in the memory 1704 and completes the steps of the aforementioned method in combination with the hardware of the processor.

[0279] The receiver 1701 may be configured to receive input digital or textual information and generate signal inputs related to relevant settings and function control of the execution device. The transmitter 1702 may be configured to output the digital or textual information via the first interface. The transmitter 1702 may further be configured to send instructions to the disk group via the first interface to modify data in the disk group. The transmitter 1702 may further include a display device, such as a display screen.

[0280] In this embodiment of the present application, the processor 1703 is optionally configured to execute an image processing method executed by an execution device in the embodiment corresponding to Figures 9 to 11. Specifically, the application processor 17031 acquires a first image; performing feature extraction on the first image to obtain at least one first feature map, the first feature map including N first feature values, where N is a positive integer; Obtain a target compression bit rate, the target compression bit rate corresponding to M target gain values, each target gain value corresponding to one first feature value, M being a positive integer less than or equal to N; processing the corresponding first feature values ​​according to the M target gain values ​​to obtain M second feature values; performing quantization and entropy coding on the at least one processed first feature map including the M second feature values ​​to obtain coded data; It is structured as follows.

[0281] Optionally, an information entropy of the quantized data obtained by quantizing the at least one processed first feature map satisfies a preset condition, the preset condition being related to a target compression bitrate.

[0282] Optionally, the pre-defined conditions include at least: This includes indicating that the higher the target compression bit rate, the greater the information entropy of the quantized data.

[0283] Optionally, the difference between the compression bit rate corresponding to the encoded data and the target compression bit rate falls within a preset range.

[0284] Optionally, the M second feature values ​​are obtained by performing multiplication operations on the M target gain values ​​and the corresponding first feature values ​​respectively.

[0285] Optionally, the at least one first feature map includes a first target feature map, the first target feature map including P first feature values, all of the P first feature values ​​corresponding to the same target gain value, where P is a positive integer less than or equal to M.

[0286] Optionally, the application processor 17031 is Determine M target gain values ​​corresponding to the target compression bitrate based on the target mapping relationship, the target mapping relationship being used to indicate a correlation between the compression bitrate and the M target gain values; the target mapping relationship includes a plurality of compressed bit rates, a plurality of gain vectors, and a correlation between the plurality of compressed bit rates and the plurality of gain vectors, the target compressed bit rate being one of the plurality of compressed bit rates, and the M target gain values ​​being one element of the plurality of gain vectors; or the target mapping relationship comprises an objective function mapping relationship, where an input of the objective function relationship comprises a target compression bit rate, and an output of the objective function relationship comprises M target gain values; It is further structured as follows.

[0287] Optionally, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponding to the M first gain values ​​and the second compression bit rate corresponding to the M second gain values, and the M target gain values ​​are obtained by performing an interpolation operation on the M first gain values ​​and the M second gain values.

[0288] Optionally, the M first gain values ​​include a first target gain value, the M second gain values ​​include a second target gain value, and the M target gain values ​​include a third target gain value, wherein the first target gain value, the second target gain value, and the third target gain value correspond to a same feature value among the M first feature values, and the third target gain value is obtained by performing an interpolation operation on the first target gain value and the second target gain value.

[0289] Optionally, the first image includes a target object, and the M first feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0290] Optionally, each of the M target gain values ​​corresponds to an inverse gain value, and the inverse gain value is used to process the feature values ​​obtained in the decoding process of the encoded data, and a product of each of the M target gain values ​​and the corresponding inverse gain value falls within a preset range.

[0291] Optionally, the application processor 17031 is performing entropy decoding on the encoded data to obtain at least one second feature map, the at least one second feature map including N third feature values, each third feature value corresponding to one first feature value; obtaining M target inverse gain values, each target inverse gain value corresponding to one third feature value; performing gain processing on the corresponding third feature values ​​based on the M target inverse gain values ​​to obtain M fourth feature values; performing image reconstruction on the at least one second feature map obtained after the inverse gain processing to obtain a second image, the at least one second feature map obtained after the inverse gain processing including the M fourth feature values; It is further structured as follows.

[0292] Optionally, the M fourth feature values ​​are obtained by performing multiplication operations on the M target inverse gain values ​​and the corresponding third feature values ​​respectively.

[0293] Optionally, the at least one second feature map includes a second target feature map, the second target feature map including P third feature values, all of the P third feature values ​​corresponding to the same target inverse gain value, where P is a positive integer less than or equal to M.

[0294] Optionally, the application processor 17031 is further configured to determine M target inverse gain values ​​corresponding to the target compression bitrate based on the target mapping relationship, the target mapping relationship being used to indicate a correlation between the compression bitrate and the inverse gain vector.

[0295] Optionally, the target mapping relationship includes a plurality of compressed bit rates, a plurality of inverse gain vectors, and a correlation relationship between the plurality of compressed bit rates and the plurality of inverse gain vectors, where the target compressed bit rate is one of the plurality of compressed bit rates, and the M target inverse gain values ​​are elements of one of the plurality of inverse gain vectors.

[0296] Optionally, the target mapping relationship comprises an objective function mapping relationship, where an input of the objective function relationship comprises the target compression bitrate and an output of the objective function relationship comprises the M target inverse gain values.

[0297] Optionally, the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0298] Optionally, the product of each of the M target gain values ​​and the corresponding target inverse gain value falls within a preset range.

[0299] Optionally, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponds to the M first inverse gain values, and the second compression bit rate corresponds to the M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values.

[0300] Optionally, the M first inverse gain values ​​include a first target inverse gain value, the M second inverse gain values ​​include a second target inverse gain value, and the M target inverse gain values ​​include a third target inverse gain value, wherein the first target inverse gain value, the second target inverse gain value, and the third target inverse gain value correspond to a same feature value among the M first feature values, and the third target inverse gain value is obtained by performing an interpolation operation on the first target inverse gain value and the second target inverse gain value.

[0301] Specifically, the application processor 17031: Obtaining the encoded data; performing entropy decoding on the encoded data to obtain at least one second feature map, the at least one second feature map including N third feature values, where N is a positive integer; Obtain M target inverse gain values, each target inverse gain value corresponding to one third feature value, where M is a positive integer less than or equal to N; respectively processing the corresponding third feature values ​​according to the M target inverse gain values ​​to obtain M fourth feature values; performing image reconstruction on the at least one processed second feature map to obtain a second image, the at least one processed second feature map including M fourth feature values; It is structured as follows.

[0302] Optionally, the M fourth feature values ​​are obtained by performing multiplication operations on the M target inverse gain values ​​and the corresponding third feature values ​​respectively.

[0303] Optionally, the at least one second feature map includes a second target feature map, the second target feature map including P third feature values, all of the P third feature values ​​corresponding to the same target inverse gain value, where P is a positive integer less than or equal to M.

[0304] Optionally, the application processor 17031 is and determining, based on a target mapping relationship, M target inverse gain values ​​corresponding to the target compression bitrate, wherein the target mapping relationship is used to indicate a correlation between the compressed bitrate and the inverse gain vector, wherein the target mapping relationship comprises a plurality of compressed bitrates, a plurality of inverse gain vectors, and correlations between the plurality of compressed bitrates and the plurality of inverse gain vectors, wherein the target compression bitrate is one of the plurality of compressed bitrates and the M target inverse gain values ​​are elements of one of the plurality of inverse gain vectors, or wherein the target mapping relationship comprises a target function mapping relationship, wherein an input of the target function relationship comprises the target compression bitrate, and an output of the target function relationship comprises the M target inverse gain values.

[0305] Optionally, the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0306] Optionally, the target compression bit rate is greater than the first compression bit rate and less than the second compression bit rate, the first compression bit rate corresponds to the M first inverse gain values, and the second compression bit rate corresponds to the M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values.

[0307] Optionally, the M first inverse gain values ​​include a first target inverse gain value, the M second inverse gain values ​​include a second target inverse gain value, and the M target inverse gain values ​​include a third target inverse gain value, wherein the first target inverse gain value, the second target inverse gain value, and the third target inverse gain value correspond to a same feature value among the M first feature values, and the third target inverse gain value is obtained by performing an interpolation operation on the first target inverse gain value and the second target inverse gain value.

[0308] An embodiment of the present application further provides a training device. FIG. 18 is a schematic diagram of the structure of a training device according to an embodiment of the present application. The image processing device described in the embodiment corresponding to FIG. 16 may be deployed in a training device 1800 to perform the functions of the image processing device in the embodiment corresponding to FIG. 16. Specifically, the training device 1800 is implemented by one or more servers. The training device 1800 may generate relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1822 (e.g., one or more processors), a memory 1832, and one or more storage media 1830 (e.g., one or more mass storage devices) that store an application 1842 or data 1844. The memory 1832 and the storage medium 1830 may be temporary or permanent storage devices. The program stored in the storage medium 1830 may include at least one module (not shown), and each module may include a series of instruction operations for the training device. Furthermore, the central processing unit 1822 is arranged to communicate with a storage medium 1830 and is capable of executing a sequence of instruction operations in the storage medium 1830 in the training device 1800 .

[0309] The training device 1800 may further include one or more power sources 1826, one or more wired or wireless network interfaces 1850, one or more input / output interfaces 1858, and / or one or more operating systems 1841, such as Windows Server™, Mac OS X™, Unix™, Linux™, or FreeBSD™.

[0310] In this embodiment of the present application, the central processing unit 1822 is configured to execute the image processing method performed by the image processing device of the embodiment corresponding to Figure 16. In particular, the central processing unit 1822: Acquire a first image; Perform feature extraction on the first image based on the encoding network to obtain at least one first feature map, where the at least one first feature map includes N first feature values, where N is a positive integer; Obtain a target compression bit rate, the target compression bit rate corresponding to M initial gain values ​​and M initial inverse gain values, each initial gain value corresponding to one first feature value, each initial inverse gain value corresponding to one third feature value, M being a positive integer less than or equal to N; processing the corresponding first feature values ​​based on the M initial gain values ​​to obtain M second feature values; Perform quantization and entropy coding on the at least one processed first feature map based on a quantization network and an entropy coding network to obtain coded data and a bit rate loss, and the at least one first feature map obtained after the gain processing includes M second feature values; performing entropy decoding on the encoded data based on the entropy decoding network to obtain at least one second feature map, the at least one second feature map including M third feature values, each third feature value corresponding to one first feature value; respectively processing the corresponding third feature values ​​according to the M initial inverse gain values ​​to obtain M fourth feature values; Perform image reconstruction on the at least one processed second feature map based on the decoding network to obtain a second image, where the at least one processed second feature map includes M fourth feature values; Obtain a distortion loss of the second image relative to the first image; Perform joint training on a first encoding / decoding network, the M initial gain values, and the M initial inverse gain values ​​by using a loss function until an image distortion value between the first image and the second image reaches a first preset degree, the image distortion value being related to a bitrate loss and a distortion loss, the encoding / decoding network includes an encoding network, a quantization network, an entropy encoding network, and an entropy decoding network; outputting a second encoding / decoding network, M target gain values, and M target inverse gain values, the second encoding / decoding network being a model obtained after iterative training is performed on the first encoding / decoding network, and the M target gain values ​​and the M target inverse gain values ​​are obtained after iterative training is performed on the M initial gain values ​​and the M initial inverse gain values; It is structured as follows.

[0311] Optionally, an information entropy of the quantized data obtained by quantizing the at least one first feature map obtained after the gain processing satisfies a preset condition, the preset condition being related to a target compression bitrate.

[0312] Optionally, the pre-defined conditions include at least: This includes indicating that the higher the target compression bit rate, the greater the information entropy of the quantized data.

[0313] Optionally, the M second feature values ​​are obtained by performing multiplication operations on the M target gain values ​​and the corresponding first feature values ​​respectively.

[0314] Optionally, the at least one first feature map includes a first target feature map, the first target feature map including P first feature values, all of the P first feature values ​​corresponding to the same target gain value, where P is a positive integer less than or equal to M.

[0315] Optionally, the first image includes a target object, and the M first feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

[0316] Optionally, the product of each of the M target gain values ​​and the corresponding target inverse gain value falls within a preset range, and the product of each of the M initial gain values ​​and the corresponding initial inverse gain value falls within a preset range.

[0317] An embodiment of the present application further provides a computer program product, which when run on a computer enables the computer to execute steps performed by an execution device in the method described in the previous embodiment shown in FIG. 17, or enables the computer to execute steps performed by a training device in the method described in the previous embodiment shown in FIG.

[0318] An embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a program for performing signal processing. When the program runs on a computer, the computer is enabled to perform the steps performed by the execution device in the method described in the previous embodiment shown in Figure 17, or the computer is enabled to perform the steps performed by the training device in the method described in the previous embodiment shown in Figure 18.

[0319] The execution device, training device, or terminal device provided in the embodiments of the present application may specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit may be, for example, a processor. The communication unit may be, for example, an input / output interface, a pin, or a circuit. The processing unit may execute computer-executable instructions stored in the storage unit to cause the chip in the execution device to perform the image processing method described in the embodiment shown in FIG. 3 to FIG. 7, or cause the chip in the training device to perform the image processing method described in the embodiment shown in FIG. 13. Optionally, the storage unit is a storage unit in the chip, for example, a register or a cache. The storage unit may alternatively be a storage unit located outside the chip at the wireless access device end, for example, a read-only memory (ROM) or another type of static storage device capable of storing static information and instructions, or a random access memory (RAM).

[0320] For details, please refer to Figure 19. Figure 19 is a schematic diagram of the structure of a chip according to an embodiment of the present application. The chip may be represented as a neural network processing unit NPU2000. The NPU2000 is mounted on a host CPU as a coprocessor, and the host CPU assigns tasks to the NPU. The core part of the NPU is an arithmetic circuit 2003, which is controlled by using a controller 2004 to extract matrix data from memory and perform multiplication operations.

[0321] In some embodiments, the arithmetic circuit 2003 includes multiple process engines (PEs). In some embodiments, the arithmetic circuit 2003 is a two-dimensional systolic array. The arithmetic circuit 2003 may be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some embodiments, the arithmetic circuit 2003 is a general-purpose matrix processor.

[0322] For example, it is assumed that there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves corresponding data of matrix B from the weight memory 2002, and buffers the data in each PE in the arithmetic circuit. The arithmetic circuit obtains data of matrix A from the input memory 2001, performs a matrix operation between the data and matrix B, and stores the partial or final result of the obtained matrix in an accumulator 2008.

[0323] The unified memory 2006 is configured to store input data and output data. The weight data is directly transferred to the weight memory 2002 by using a Direct Memory Access Controller (DMAC) 2005. The input data is also transferred to the unified memory 2006 by using the DMAC.

[0324] The BIU is a Bus Interface Unit, ie, a Bus Interface Unit 2010, configured to interact with a DMAC and an Instruction Fetch Buffer (IFB) 2009 by using an AXI bus.

[0325] The Bus Interface Unit 2010 (BIU for short) is further configured such that the instruction fetch buffer 2009 is configured to retrieve instructions from an external memory, and the direct memory access controller 2005 is further configured to retrieve raw data of the input matrix A or the weighting matrix B from the external memory.

[0326] The DMAC is mainly configured to transfer input data in the external memory DDR to the unified memory 2006, transfer weight data to the weight memory 2002, or transfer input data to the input memory 2001.

[0327] The vector calculation unit 2007 includes multiple arithmetic processing units. If necessary, further processing is performed on the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithm operation, value comparison, etc. The vector calculation unit 2007 is mainly configured to perform network calculations, such as batch normalization, pixel-level addition, and feature surface upsampling, for non-convolutional / fully connected layers in the neural network.

[0328] In some implementations, the vector computation unit 2007 may store the processed output vector in the unified memory 2006. For example, the vector computation unit 2007 may apply a linear and / or non-linear function to the output of the computation circuitry 2003, e.g., perform linear interpolation on the feature surface extracted in the convolutional layer. In another example, the linear and / or non-linear function is applied to the vector of accumulated values ​​to generate activation values. In some implementations, the vector computation unit 2007 generates a normalization value, a pixel-level sum, or a normalization value and a pixel-level sum. In some implementations, the processed output vector may be used as an activated input to the computation circuitry 2003, e.g., the processed output vector may be used in a subsequent layer of a neural network.

[0329] An instruction fetch buffer 2009 coupled to the controller 2004 is configured to store instructions used by the controller 2004 .

[0330] The unified memory 2006, the input memory 2001, the weight memory 2002, and the instruction fetch buffer 2009 are all on-chip memories. The external memory is dedicated to the hardware architecture of the NPU.

[0331] The processor referred to in any of the above may be a general purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits configured to control program execution of the method according to the first aspect.

[0332] In addition, it should be noted that the described device embodiments are merely examples. Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, and may be located in one place or distributed over multiple network units. Some or all modules may be selected according to actual requirements to achieve the purpose of the solution of the embodiment. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that the modules have communication connections with each other, which may be specifically implemented as one or more communication buses or signal cables.

[0333] Based on the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by software in addition to the necessary general-purpose hardware, or can be implemented by dedicated hardware including, of course, application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, any function that can be performed by a computer program can be easily implemented by using corresponding hardware, and the specific hardware structure used to achieve the same function may be in various forms, for example, in the form of an analog circuit, a digital circuit, or a dedicated circuit. However, in the present application, the implementation of a software program is in most cases a better implementation. Based on such understanding, the technical solution of the present application may be essentially or the part that contributes to the prior art may be implemented in the form of a software product. The software product is stored in a readable storage medium such as a computer floppy disk, a USB flash drive, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, and includes some instructions for instructing a computer device (which may be a personal computer, a training device, or a network device) to execute the method described in the embodiment of the present application.

[0334] All or part of the above-described embodiments may be implemented using software, hardware, firmware, or any combination thereof. If software is used to implement the embodiments, all or part of the embodiments may be implemented in the form of a computer program product.

[0335] A computer program product includes one or more computer instructions. When the computer program instructions are loaded into a computer and executed, all or part of the procedures or functions according to the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, training device, or data center to another website, computer, training device, or data center in a wired (e.g., coaxial cable, optical fiber, or digital subscriber line (DSL)) or wireless (e.g., infrared, radio frequency, or microwave) manner. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device that integrates one or more available media, such as a training device or a data center. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), a semiconductor medium (e.g., a solid state disk (SSD)), etc. [Explanation of symbols]

[0336] 200 Image Processing System 201 Target Model / Rules 210 Execution device 211 Calculation Module 212 I / O Interface 220 Training equipment 230 Database 240 Client Device 250 Data Storage System 401 1st image 402 CNN / layer 403 Multi-Channel-Wise Feature Maps 1400 Image Processing Device 1401 Acquisition Module 1402 Feature Extraction Module 1403 Gain Module 1404 Entropy Coding Module 1500 Image Processing Device 1501 Acquisition Module 1502 Decryption Module 1503 Inverse Gain Module 1504 Reconstruction Module 1600 Image Processing Device 1601 Acquisition Module 1602 Feature Extraction Module 1603 Gain Module 1604 Entropy Coding Module 1605 Decryption Module 1606 Inverse Gain Module 1607 Reconstruction Module 1608 Training Module 1609 Output Module 1700 Execution device 1701 Receiver 1702 Transmitter 1703 Processor 1704 Memory 1800 training equipment 1822 Central Processing Unit 1826 power supply 1830 storage medium 1832 Memory 1841 Operating Systems 1842 Applications 1844 Data 1850 Wireless Network Interface 1858 Output Interface 2000 Neural Network Processing Unit (NPU) 2001 Input Memory 2002 Weight Memory 2003 Arithmetic circuit 2004 Controller 2005 Direct Memory Access Controller 2006 Unified Memory 2007 Vector Calculation Unit 2008 Accumulator 2009 Instruction Fetch Buffer 2010 Bus Interface Unit 17031 Application Processor 17032 Communications Processor

Claims

1. 1. An image processing method, comprising: acquiring a first image; performing feature extraction on the first image to obtain at least one first feature map, the at least one first feature map comprising N first feature values, where N is a positive integer; obtaining a target compression bit rate, the target compression bit rate corresponding to M target gain values, each target gain value corresponding to one first feature value, where M is a positive integer less than or equal to N; processing each corresponding first feature value based on the M target gain values ​​to obtain M second feature values; performing quantization and entropy coding on at least one processed first feature map to obtain encoded data, the at least one processed first feature map including the M second feature values; An image processing method comprising:

2. The method of claim 1 , wherein an information entropy of quantized data obtained by quantizing the at least one processed first feature map satisfies a preset condition, the preset condition being related to the target compression bit rate.

3. The preset conditions include at least The method of claim 2 , wherein a higher target compression bit rate indicates a greater information entropy of the quantized data.

4. The method according to claim 1 , wherein a difference between a compression bit rate corresponding to the encoded data and the target compression bit rate falls within a preset range.

5. The method according to claim 1 , wherein the M second feature values ​​are obtained by performing multiplication operations on the M target gain values ​​and the corresponding first feature values ​​respectively.

6. 6. The method of claim 1, wherein the at least one first feature map comprises a first target feature map, the first target feature map comprising P first feature values, all of the P first feature values ​​corresponding to a same target gain value, where P is a positive integer less than or equal to M.

7. The method further comprising: determining the M target gain values ​​corresponding to the target compression bitrate based on a target mapping relationship, the target mapping relationship being used to indicate a correlation between a compression bitrate and the M target gain values; the target mapping relationship comprises a plurality of compressed bit rates, a plurality of gain vectors, and a correlation between the plurality of compressed bit rates and the plurality of gain vectors, the target compressed bit rate being one of the plurality of compressed bit rates, and the M target gain values ​​being elements of one of the plurality of gain vectors; or the target mapping relationship comprises an objective function mapping relationship, and where an input of the objective function relationship comprises the target compression bit rate, an output of the objective function relationship comprises the M target gain values.

7. The method according to any one of claims 1 to 6.

8. 8. The method of claim 1, wherein the target compression bit rate is greater than a first compression bit rate and less than a second compression bit rate, the first compression bit rate corresponds to M first gain values, the second compression bit rate corresponds to M second gain values, and the M target gain values ​​are obtained by performing an interpolation operation on the M first gain values ​​and the M second gain values.

9. 9. The method of claim 1, wherein the first image includes a target object, and the M first feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

10. 10. The method according to claim 1, wherein each of the M target gain values ​​corresponds to an inverse gain value, the inverse gain value being used to process feature values ​​obtained in a decoding process of the encoded data, and a product of each of the M target gain values ​​and the corresponding inverse gain value falls within a preset range.

11. 1. An image processing method, comprising: obtaining encoded data; performing entropy decoding on the encoded data to obtain at least one second feature map, the at least one second feature map comprising N third feature values, where N is a positive integer; obtaining M target inverse gain values, each target inverse gain value corresponding to one third feature value, where M is a positive integer less than or equal to N; processing each corresponding third feature value based on the M target inverse gain values ​​to obtain M fourth feature values; performing image reconstruction on at least one processed second feature map to obtain a second image, the at least one processed second feature map including the M fourth feature values; An image processing method comprising:

12. The method of claim 11 , wherein the M fourth feature values ​​are obtained by performing multiplication operations on the M target inverse gain values ​​and the corresponding third feature values, respectively.

13. 13. The method of claim 11 or 12, wherein the at least one second feature map comprises a second target feature map, the second target feature map comprising P third feature values, all of the P third feature values ​​corresponding to a same target inverse gain value, where P is a positive integer less than or equal to M.

14. The method further comprising: obtaining a target compression bit rate; determining the M target inverse gain values ​​corresponding to the target compression bit rate based on a target mapping relationship, the target mapping relationship being used to indicate a correlation between a compression bit rate and an inverse gain vector; the target mapping relationship comprises a plurality of compressed bit rates, a plurality of inverse gain vectors, and a correlation between the plurality of compressed bit rates and the plurality of inverse gain vectors, the target compressed bit rate being one of the plurality of compressed bit rates, and the M target inverse gain values ​​being elements of one of the plurality of inverse gain vectors; or the target mapping relationship comprises an objective function mapping relationship, and where an input of the objective function relationship comprises the target compression bit rate, an output of the objective function relationship comprises the M target inverse gain values.

14. The method according to any one of claims 11 to 13.

15. 15. The method of claim 11, wherein the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

16. 16. The method of claim 11, wherein the target compression bit rate is greater than a first compression bit rate and less than a second compression bit rate, the first compression bit rate corresponding to M first inverse gain values ​​and the second compression bit rate corresponding to M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values.

17. An image processing device, an acquisition module configured to acquire a first image; a feature extraction module configured to perform feature extraction on the first image to obtain at least one first feature map, the at least one first feature map comprising N first feature values, where N is a positive integer; The obtaining module is further configured to obtain a target compression bit rate, the target compression bit rate corresponding to M target gain values, each target gain value corresponding to one first feature value, M being a positive integer less than or equal to N; a gain module configured to process each corresponding first feature value based on the M target gain values ​​to obtain M second feature values; a quantization and entropy coding module configured to perform quantization and entropy coding on at least one processed first feature map to obtain coded data, the at least one processed first feature map including the M second feature values; An image processing device comprising:

18. 20. The apparatus of claim 17, wherein an information entropy of quantized data obtained by quantizing the at least one processed first feature map satisfies a preset condition, the preset condition being related to the target compression bit rate.

19. The preset conditions include at least 20. The apparatus of claim 18, wherein a higher target compression bit rate indicates greater information entropy of the quantized data.

20. 20. The apparatus of claim 17, wherein a difference between a compression bit rate corresponding to the encoded data and the target compression bit rate falls within a preset range.

21. 21. The apparatus according to claim 17, wherein the M second feature values ​​are obtained by performing multiplication operations on the M target gain values ​​and the corresponding first feature values ​​respectively.

22. 22. The apparatus of claim 17, wherein the at least one first feature map comprises a first target feature map, the first target feature map comprising P first feature values, all of the P first feature values ​​corresponding to a same target gain value, where P is a positive integer less than or equal to M.

23. The apparatus, a determining module for determining the M target gain values ​​corresponding to the target compression bitrate based on a target mapping relationship, the target mapping relationship being used to indicate a correlation between a compression bitrate and the M target gain values; the target mapping relationship comprises a plurality of compressed bit rates, a plurality of gain vectors, and a correlation between the plurality of compressed bit rates and the plurality of gain vectors, the target compressed bit rate being one of the plurality of compressed bit rates, and the M target gain values ​​being elements of one of the plurality of gain vectors; or the target mapping relationship comprises an objective function mapping relationship, and where an input of the objective function relationship comprises the target compression bit rate, an output of the objective function relationship comprises the M target gain values.

23. Apparatus according to any one of claims 17 to 22.

24. 24. The apparatus of claim 17, wherein the target compression bit rate is greater than a first compression bit rate and less than a second compression bit rate, the first compression bit rate corresponding to M first gain values ​​and the second compression bit rate corresponding to M second gain values, and the M target gain values ​​are obtained by performing an interpolation operation on the M first gain values ​​and the M second gain values.

25. 25. The apparatus of claim 17, wherein the first image includes a target object, and the M first feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

26. 26. The apparatus of claim 17, wherein each of the M target gain values ​​corresponds to an inverse gain value, the inverse gain value being used to process feature values ​​obtained in a decoding process of the encoded data, and a product of each of the M target gain values ​​and the corresponding inverse gain value falls within a preset range.

27. An image processing device, a capture module configured to capture the encoded data; a decoding module configured to perform entropy decoding on the encoded data to obtain at least one second feature map, the at least one second feature map comprising N third feature values, where N is a positive integer; the acquisition module is further configured to acquire M target inverse gain values, each target inverse gain value corresponding to one third feature value, where M is a positive integer less than or equal to N; an inverse gain module configured to process each corresponding third feature value based on the M target inverse gain values ​​to obtain M fourth feature values; a reconstruction module configured to perform image reconstruction on the at least one processed second feature map to obtain a second image, the at least one processed second feature map including the M fourth feature values; An image processing device comprising:

28. 28. The apparatus of claim 27, wherein the M fourth feature values ​​are obtained by performing multiplication operations on the M target inverse gain values ​​and the corresponding third feature values, respectively.

29. 29. The apparatus of claim 27 or 28, wherein the at least one second feature map comprises a second target feature map, the second target feature map comprising P third feature values, all of the P third feature values ​​corresponding to a same target inverse gain value, where P is a positive integer less than or equal to M.

30. the capture module is further configured to capture a target compression bit rate; The apparatus, a determining module for determining the M target inverse gain values ​​corresponding to the target compression bitrate based on a target mapping relationship, the target mapping relationship being used to indicate a correlation between a compression bitrate and an inverse gain vector; the target mapping relationship comprises a plurality of compressed bit rates, a plurality of inverse gain vectors, and a correlation between the plurality of compressed bit rates and the plurality of inverse gain vectors, the target compressed bit rate being one of the plurality of compressed bit rates, and the M target inverse gain values ​​being elements of one of the plurality of inverse gain vectors; or the target mapping relationship comprises an objective function mapping relationship, and where an input of the objective function relationship comprises the target compression bit rate, an output of the objective function relationship comprises the M target inverse gain values.

30. Apparatus according to any one of claims 27 to 29.

31. 31. The apparatus of claim 27, wherein the second image includes a target object, and the M third feature values ​​are feature values ​​in the at least one feature map that correspond to the target object.

32. 32. The apparatus of claim 27, wherein the target compression bit rate is greater than a first compression bit rate and less than a second compression bit rate, the first compression bit rate corresponding to M first inverse gain values ​​and the second compression bit rate corresponding to M second inverse gain values, and the M target inverse gain values ​​are obtained by performing an interpolation operation on the M first inverse gain values ​​and the M second inverse gain values.

33. 17. An image processing apparatus comprising a non-volatile memory and a processor coupled to each other, the processor calling program code stored in the memory to perform the method of any one of claims 1 to 16.

Images