Image processing method and device, storage medium and electronic device

Abstract

This invention discloses an image processing method and apparatus, a storage medium, and an electronic device. The method includes: acquiring a first set of pixel blocks to be encoded in a first image; encoding each pixel block in the first set of pixel blocks to obtain a first set of encoded data; decoding the first set of encoded data to obtain a second set of pixel blocks, and acquiring a first set of decoded information during the decoding of the first set of encoded data; inputting the first set of decoded information into a target attention network model to obtain a target weight value set; inputting the second set of pixel blocks and the target weight value set into a target neural network model to obtain a third set of pixel blocks output by the target neural network model, wherein the target neural network model is used to adjust the pixel values ​​of the pixels in the second set of pixel blocks according to the target weight value set to obtain the third set of pixel blocks, and the third set of pixel blocks is used to form a second image.

Description

Technical Field

[0001] This invention relates to the field of communications, and more specifically, to an image processing method and apparatus, a storage medium, and an electronic device. Background Technology

[0002] With the rapid development of the video imaging industry, video resolution has increased from standard definition, high definition, ultra-high definition to 4K / 8K, and frame rates have increased from 30, 60, 90 to 120 frames per second. The amount of information contained in these videos is constantly expanding, inevitably putting immense pressure on network bandwidth. Therefore, improving video image quality has become crucial. There are two methods: one is to continuously increase the transmission bitrate, and the other is to perform image processing before display. Increasing the bitrate means increasing bandwidth, which increases costs and obviously cannot be increased indefinitely. Therefore, image processing at the display end can be adjusted according to the scenario, maximizing display quality with minimal cost increase.

[0003] To improve coding quality, the international video coding standards organization proposed the HEVC standard, also known as H.265, which introduces a Coding Tree Unit (CTU) and employs a quadtree-based image block partitioning method. This block partitioning method can adaptively divide the image based on texture information. The size of the Coding Unit (CU) ranges from a minimum of 8x8 to a maximum of 64x64. The advantages of this partitioning are twofold: firstly, larger CUs can significantly improve coding efficiency in smooth areas; secondly, smaller CUs can better handle local image details, thus enabling more accurate prediction of complex images.

[0004] In the typical coding process, different regions are divided into intra-frame coding units and inter-frame coding units. Intra-frame blocks only consider spatial information, while inter-frame blocks only consider temporal information. After prediction, transform quantization is required to better identify which pixels have rich textures but are difficult to predict, i.e., residuals need to be encoded.

[0005] Generally speaking, according to the bitrate control algorithm, regions with rich texture information tend to choose smaller quantization parameters (qp) when encoding, while flat regions tend to choose larger quantization values, in order to reduce the bitstream consumed during the encoding process.

[0006] Compared to traditional image processing, deep learning and neural networks can greatly improve the subjective and objective quality of video images. These include SRCNN, DRCN, ESPCN, VESPCN, and SRGAN, which significantly improve the quality of video images obtained by traditional image processing methods. Therefore, combining deep learning and neural networks in image processing is a future development trend.

[0007] The utilization of detailed information has a significant impact on image processing performance. Correctly and fully utilizing this information can improve the final results of image processing. Currently, video image processing only utilizes the data information of the video image itself. If image processing occurs after decoding, the decoded information can be used to further improve deep neural network learning, more effectively extracting and recovering texture information.

[0008] In related technologies, traditional image processing based on neural network algorithms only utilizes the image data obtained after decoding the encoded stream, but does not utilize the decoded information obtained after decoding the encoded stream, resulting in poor image processing performance. Currently, no effective solution has been proposed.

[0009] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0010] This invention provides an image processing method, apparatus, storage medium, and electronic device to at least solve the problem that traditional image processing based on neural network algorithms only utilizes the image data obtained after decoding the encoded stream, but does not utilize the decoded information obtained after decoding the encoded stream, thus resulting in poor image processing performance.

[0011] According to one aspect of the present invention, an image processing method is provided, comprising: acquiring a first set of pixel blocks to be encoded in a first image; encoding each pixel block in the first set of pixel blocks to obtain a first set of encoded data, wherein each pixel block corresponds to an encoding method; decoding the first set of encoded data to obtain a second set of pixel blocks, and acquiring a first set of decoding information when decoding the first set of encoded data, wherein the decoding information in the first set of decoding information corresponds one-to-one with each pixel block in the second set of pixel blocks; inputting the first set of decoding information into a target attention network model to obtain a target set of weight values, wherein the target set of weight values ​​includes weight values ​​corresponding one-to-one with each pixel in the second set of pixel blocks, the weight values ​​corresponding to each pixel being used to represent the importance of each pixel in the second set of pixel blocks; inputting the second set of pixel blocks and the target set of weight values ​​into a target neural network model to obtain a third set of pixel blocks output by the target neural network model, wherein the target neural network model is used to adjust the pixel values ​​of the pixels in the second set of pixel blocks according to the target set of weight values ​​to obtain the third set of pixel blocks, the third set of pixel blocks being used to form a second image.

[0012] According to another aspect of the present invention, an image processing apparatus is also provided, comprising: a first acquisition module, configured to acquire a first set of pixel blocks to be encoded in a first image, and encode each pixel block in the first set of pixel blocks to obtain a first encoded data set, wherein each pixel block corresponds to an encoding method; a decoding module, configured to decode the first encoded data set to obtain a second set of pixel blocks, and acquire a first decoding information set when decoding the first set of encoded data, wherein the decoding information in the first decoding information set corresponds one-to-one with each pixel block in the second set of pixel blocks; and a second acquisition module, configured to input the first decoding information set into a target attention network. The model obtains a target weight value set, wherein the target weight value set includes a weight value corresponding one-to-one with each pixel in the second pixel block set, and the weight value corresponding to each pixel is used to represent the importance of each pixel in the second pixel block set; a third acquisition module is used to input the second pixel block set and the target weight value set into the target neural network model to obtain a third pixel block set output by the target neural network model, wherein the target neural network model is used to adjust the pixel values ​​of the pixels in the second pixel block set according to the target weight value set to obtain the third pixel block set, and the third pixel block set is used to form a second image.

[0013] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which is configured to execute the above-described image processing method at runtime.

[0014] According to another aspect of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the image processing method described above through the computer program.

[0015] This invention, during the transmission of a first image, encodes each pixel block in the first pixel block set to be encoded in the first image to obtain a first encoded data set. Then, at the receiving end, the obtained first encoded data set is decoded to obtain a second pixel block set, and a first decoding information set is obtained during the decoding of the first encoded data set. This first decoding information set is input into a target attention network model to obtain a target weight value set. Finally, the second pixel block set and the target weight value set are input into a target neural network model to obtain a third pixel block set output by the target neural network model. The third pixel block set is used to form the second image. Using this technical solution, for each pixel block in the image, the target neural network model can determine the importance of pixels in the pixel block based on the decoding information, thus focusing on processing important pixels and improving the image processing effect. This solves the problem that traditional image processing based on neural network algorithms only utilizes the image data obtained after decoding the encoded stream, without utilizing the decoding information obtained after decoding the encoded stream, resulting in poor image processing performance. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with the description thereof, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0017] Figure 1 This is a hardware structure block diagram of a computer terminal for the image processing method according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0019] Figure 3 This is a framework diagram (a) of an image processing method according to an embodiment of the present invention;

[0020] Figure 4 This is a framework diagram (II) of an image processing method according to an embodiment of the present invention;

[0021] Figure 5 This is a framework diagram (III) of an image processing method according to an embodiment of the present invention;

[0022] Figure 6 This is a framework diagram (four) of an image processing method according to an embodiment of the present invention;

[0023] Figure 7 This is a frame diagram (a) of an image processing apparatus according to an embodiment of the present invention;

[0024] Figure 8This is a frame diagram (II) of an image processing apparatus according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for the image processing method according to an embodiment of the present invention. Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor unit (MPU) or a programmable logic device (PLD)) and a memory 104 for storing data are also shown. In one exemplary embodiment, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0028] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the image processing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0030] This embodiment provides an image processing method applied to a target client. Figure 2 This is a flowchart of an image processing method according to an embodiment of the present invention, which includes the following steps:

[0031] Step S202: Obtain the first set of pixel blocks to be encoded in the first image, encode each pixel block in the first set of pixel blocks to obtain the first set of encoded data, wherein each pixel block corresponds to an encoding method;

[0032] It should be noted that the first image needs to be compressed during transmission. During the compression process, the first image can be divided into multiple pixel blocks, which in turn form a set of first pixel blocks. The encoding method corresponding to the pixel blocks can be one of the following: intra-frame encoding method or inter-frame encoding method.

[0033] Step S204: Decode the first encoded data set to obtain the second pixel block set, and obtain the first decoding information set when decoding the first encoded data set, wherein the decoding information in the first decoding information set corresponds one-to-one with each pixel block in the second pixel block set;

[0034] Step S206: Input the first decoded information set into the target attention network model to obtain the target weight value set, wherein the target weight value set includes a weight value that corresponds one-to-one with each pixel in the second pixel block set, and the weight value corresponding to each pixel is used to represent the importance of each pixel in the second pixel block set;

[0035] To better understand this, let's assume we have an image where the main subject is a person and the background is grassland. Then we can understand that the pixels of the person in the image are more important than the pixels of the grassland.

[0036] It's worth noting that attention models are widely used in various deep learning tasks, including natural language processing, image recognition, and speech recognition. They are one of the most noteworthy and in-depth core technologies in deep learning. The core idea of ​​attention is to utilize a similarity metric: when the current input is similar to the target state, its weight is increased to capture more detailed information about the target while suppressing other irrelevant information.

[0037] Step S208: Input the second pixel block set and the target weight value set into the target neural network model to obtain the third pixel block set output by the target neural network model. The target neural network model is used to adjust the pixel values ​​of the pixels in the second pixel block set according to the target weight value set to obtain the third pixel block set. The third pixel block set is used to form the second image.

[0038] It should be noted that the purpose of the image processing in the above embodiments is to make the pixel values ​​of the pixels in the second image closer to the pixel values ​​of the pixels in the first image.

[0039] It should be noted that the first set of pixel blocks together constitutes the first image, which is the original image before transmission without compression. To improve transmission efficiency, the first image is compressed before transmission. An encoded stream is obtained. After decoding the encoded stream at the receiving end, a second set of pixel blocks is obtained. Because the compression process is lossy, the pixel blocks in the second set are different from those in the first set. To ensure that the image quality of the second image displayed at the receiving end is close to that of the first image, image processing is required for each pixel block in the second set. First, the decoded information obtained after decoding the encoded stream is input into an attention network model to obtain a target weight value set. Then, the second set of pixel blocks and the target weight value set are input into a target neural network model to obtain a third set of pixel blocks. This third set of pixel blocks forms the second image, which is then displayed at the receiving end.

[0040] Through the above steps, during the transmission of the first image, each pixel block in the first pixel block set to be encoded in the first image is encoded to obtain a first encoded data set. Then, at the receiving end, the obtained first encoded data set is decoded to obtain a second pixel block set, and a first decoding information set is obtained when decoding the first encoded data set. This first decoding information set is input into a target attention network model to obtain a target weight value set. Then, the second pixel block set and the target weight value set are input into a target neural network model to obtain a third pixel block set output by the target neural network model. The third pixel block set is used to form the second image. Using this technical solution, for each pixel block in the image, the target neural network model can determine the importance of pixels in the pixel block based on the decoding information, and thus focus on processing important pixels, improving the image processing effect. This solves the problem that traditional image processing based on neural network algorithms only utilizes the image data obtained after decoding the encoded stream, without utilizing the decoding information obtained after decoding the encoded stream, resulting in poor image processing performance.

[0041] To better understand the above-mentioned acquisition of the first decoding information set when decoding the first encoded data set, in an optional embodiment, it can be implemented through the following technical solution: when the encoding method corresponding to the first pixel block is intra-frame coding, the first decoding information when decoding the first encoded data is acquired, wherein the first pixel block set includes the first pixel block, the first encoded data is encoded data obtained by encoding the first pixel block using the intra-frame coding method, and the first decoding information includes the size information of the first pixel block and a first quantization parameter, wherein the first quantization parameter is the quantization parameter used to encode the first pixel block; and / or when the encoding method corresponding to the second pixel block is inter-frame coding, the second decoding information when decoding the second encoded data is acquired, wherein the first pixel block set includes the second pixel block, the second encoded data is encoded data obtained by encoding the second pixel block using the inter-frame coding method, and the second decoding information includes a residual information set and a second quantization parameter, wherein the second quantization parameter is the quantization parameter used to encode the second pixel block, and the residual information set includes the residual information corresponding to each pixel in the second pixel block.

[0042] In other words, assuming the first pixel block set contains two distinct pixel blocks, namely the first pixel block and the second pixel block, and the first pixel block uses intra-frame coding while the second pixel block uses inter-frame coding, then for the first pixel block, when the decoder decodes the encoded stream of the first pixel block, it will obtain the corresponding third pixel block, as well as decoding information about the third pixel block. It should be noted that due to lossy compression during transmission, the pixel values ​​of corresponding pixels in the third pixel block and the first pixel block are different. Since the first pixel block uses intra-frame coding, it is only necessary to obtain the pixel block size information and the first quantization parameter from the decoding information about the third pixel block.

[0043] For the second pixel block, when the decoder decodes the encoded stream of the second pixel block, it obtains the corresponding fourth pixel block, along with decoding information about the fourth pixel block. It's important to note that due to lossy compression during transmission, the pixel values ​​of corresponding pixels in the fourth and second pixel blocks are different. Since the first pixel block uses inter-frame coding, only the second quantization parameter and residual information set from the decoding information of the fourth pixel block are needed. The residual information set contains the residual value of each pixel in the fourth pixel block. To better understand the residual value, let's assume a pixel A in the second pixel block has a pixel value of 200. After encoding, the predicted pixel value is 50. The actual residual value of pixel A is 150. However, due to lossy compression, the residual value of pixel A is also compressed. Therefore, in the fourth pixel block obtained after decoding the encoded stream of the second pixel block, the pixel value of pixel A is 50, and the decoded deviation residual value may also be 50. In this embodiment, the residual value of each pixel in the residual information set is the deviation residual value.

[0044] In an optional embodiment, the above-mentioned inputting the first decoded information set into the target attention network model to obtain the target weight value set includes the following method: when the encoding method corresponding to the first pixel block is the intra-frame encoding method, the first decoded information is input into the target attention network model to obtain the first weight value set, wherein the first weight value set includes weight values ​​corresponding one-to-one with each pixel in the third pixel block, the target weight value set includes the first weight set, the second pixel block set includes the third pixel block, the third pixel block is a pixel block obtained by decoding the first encoded data of the first pixel block, and the target attention model is used to determine the first weight set of the third pixel block according to the product of the size information of the first pixel block and the first weight, and the product of the first quantization parameter and the second weight.

[0045] In other words, for the first pixel block, since it is an intra-frame encoded block, the first decoded information obtained after decoding the encoded stream of the first pixel block is input into the target attention network model. The target attention network model then determines the first weight set of the third pixel block based on the product of the pixel block size information and the first weight, and the product of the first quantization parameter and the second weight. It should be noted that during encoding, for regions with rich texture information within a pixel block, the size of the encoding unit is smaller, and a smaller quantization value is used. If there are two pixel blocks, where pixel block A is smaller than pixel block B, and the quantization parameter used for pixel block A is also smaller than that used for pixel block B, then the weight values ​​of the pixels in pixel block A are larger than those in pixel block B. Specifically, the weight value of a pixel block can be determined by taking the reciprocal of the sum of the product of the first pixel block size information and the first weight, and the product of the first quantization parameter and the second weight. Since the pixels in each pixel block corresponding to the intra-frame encoded block have the same quantization value and are located in the same pixel block (the pixel block size is the same), the weight value of each pixel is the same. It should be noted that in an image, if there is a region where the pixel values ​​of the pixels differ significantly, then this region is considered to have rich texture information.

[0046] In an optional embodiment, the above-mentioned inputting the first decoded information set into the target attention network model to obtain the target weight value set may further include the following: when the encoding method corresponding to the second pixel block is the inter-frame encoding method, the second decoded information is input into the target attention network model to obtain the second weight value set, wherein the second weight value set includes weight values ​​corresponding one-to-one with each pixel in the fourth pixel block, the target weight value set includes the second weight set, the second pixel block set includes the fourth pixel block, the fourth pixel block is a pixel block obtained after decoding the second encoded data of the second pixel block, and the target attention model is used to determine the second weight set of the fourth pixel block based on the product of the second quantization parameter and the third weight, and the product of the residual information set and the fourth weight.

[0047] In other words, for the second pixel block, since it uses inter-frame coding, the second decoded information obtained after decoding the encoded stream of the second pixel block is input into the target attention network model. This allows the target attention network model to determine the second weight set of the fourth pixel block based on the product of the second quantization parameter and the third weight in the second decoded information, and the product of the residual information set and the fourth weight. It should be noted that during encoding, regions with rich texture information within a pixel block have smaller residuals and use smaller quantization values. If a pixel block contains pixel A and pixel B, since the quantization parameters of pixel A and pixel B are the same, it is only necessary to compare the residuals of pixel A and pixel B. If the residual of pixel A is smaller than the residual of pixel B, then the weight value of pixel A is larger than the weight value of pixel B. Specifically, the weight value of each pixel block can be determined by taking the product of the second quantization parameter and the third weight, and the reciprocal of the sum of the products of the residual values ​​of each pixel in the residual information set and the second weight.

[0048] Then, by adopting the above technical solution, the weight value of each pixel in the second pixel block set is determined, and thus they together constitute the target weight value set.

[0049] In an optional embodiment, the step of inputting the second pixel block set and the target weight value set into the target neural network model to obtain the third pixel block set output by the target neural network model can be achieved by: inputting the fifth pixel block in the second pixel block set and the fifth weight value set corresponding to the fifth pixel block in the target weight value set into the target neural network model to obtain the sixth pixel block in the third pixel block set. The target neural network model is used to adjust the pixel values ​​of the pixels in the fifth pixel block according to adjustment parameters corresponding to the weight values ​​in the fifth weight value set to obtain the sixth pixel block. The adjustment parameters corresponding to the fifth weight value set include the allowed adjustment range and / or allowed adjustment precision of the pixel values ​​when adjusting the pixel values ​​of the pixels in the fifth pixel block.

[0050] To better understand, let's take any pixel block in the second pixel block set, the fifth pixel block, as an example. The fifth pixel block can be coded intra-frame or inter-frame. The fifth pixel block and its corresponding weight value are input into the target neural network model. The target neural network model then adjusts the pixel values ​​of the pixels in the fifth pixel block according to the adjustment parameters corresponding to each weight value in the weight value set. Specifically, the adjustment range and precision of the pixel values ​​need to be determined based on the weight values.

[0051] In one exemplary embodiment, the adjustment range [-20, 20] is greater than the adjustment range [-10, 10], and the adjustment precision of 3 pixel values ​​is greater than the adjustment precision of 4 pixel values.

[0052] To better understand how to determine the adjustment parameter corresponding to each weight value in the fifth set of weight values, one of the following methods can be used: determine the adjustment parameter corresponding to each weight value in the fifth set of weight values ​​based on the ranking of each weight value in the target set of weight values; determine the adjustment parameter corresponding to each weight value in the fifth set of weight values ​​based on the preset range in which each weight value in the fifth set of weight values ​​is located.

[0053] In an optional embodiment, the adjustment parameter corresponding to each weight value in the fifth weight value set is determined based on the weight value ranking of each weight value in the fifth weight value set. This can be determined in the following way: if the weight value ranking of the first weight value in the fifth weight value set is within a first range, the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the first range; if the weight value ranking of the second weight value in the fifth weight value set is within a second range, the second adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the second range; wherein, each of the multiple ranges including the first range and the second range is pre-configured with a corresponding adjustment parameter, the fifth weight value set includes the first weight value and the second weight value, each ranking in the first range is higher than each ranking in the second range, and the first adjustment parameter... The adjustment range of the first adjustment parameter is greater than the adjustment range of the second adjustment parameter and / or the adjustment precision of the first adjustment parameter is higher than the adjustment precision corresponding to the second adjustment parameter; or, if the weight value of the first weight value ranks higher than the weight value of the second weight value in the fifth weight value set, the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the weight value ranking of the first weight value in the fifth weight value set, and the second adjustment parameter corresponding to the second weight value is determined to be equal to the adjustment parameter corresponding to the weight value ranking of the second weight value in the fifth weight value set, wherein each weight value ranking in the fifth weight value set is pre-configured with a corresponding adjustment parameter, and the adjustment range of the first adjustment parameter is greater than the adjustment range of the second adjustment parameter and / or the adjustment precision of the first adjustment parameter is higher than the adjustment precision corresponding to the second adjustment parameter.

[0054] To better understand, the following is a detailed explanation: If the first weight value of pixel A ranks third in the fifth set of weight values ​​and falls within the first range (first to fifth), then the first adjustment parameter pre-configured within the first range is determined as the adjustment parameter for pixel A. If the second weight value of pixel B ranks sixth in the fifth set of weight values ​​and falls within the second range (sixth to tenth), then the second adjustment parameter pre-configured within the second range is determined as the adjustment parameter for pixel B.

[0055] In this implementation, an adjustment parameter can be preset for each ranking in the fifth set of weight values, and the adjustment parameter corresponding to each weight value can be determined to be equal to the adjustment parameter corresponding to the weight value ranking of each weight value in the fifth set of weight values.

[0056] In an optional embodiment, determining the adjustment parameter corresponding to each weight value in the fifth weight value set based on the weight value ranking of each weight value in the target weight value set can be achieved in the following way: when the weight value ranking of the first weight value in the target weight value set is within a first range, the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the first range; when the weight value ranking of the second weight value in the target weight value set is within a second range, the second adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the second range; wherein, each of the multiple ranges including the first range and the second range is pre-configured with a corresponding adjustment parameter, the fifth weight value set includes the first weight value and the second weight value, each ranking in the first range is higher than each ranking in the second range, and the first adjustment parameter... The adjustment range of the first adjustment parameter is greater than the adjustment range of the second adjustment parameter and / or the adjustment precision of the first adjustment parameter is higher than the adjustment precision corresponding to the second adjustment parameter; or, if the weight value of the first weight value ranks higher than the weight value of the second weight value in the target weight value set, the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the weight value ranking of the first weight value in the target weight value set, and the second adjustment parameter corresponding to the second weight value is determined to be equal to the adjustment parameter corresponding to the weight value ranking of the second weight value in the target weight value set, wherein each weight value ranking in the target weight value set is pre-configured with a corresponding adjustment parameter, and the adjustment range of the first adjustment parameter is greater than the adjustment range of the second adjustment parameter and / or the adjustment precision of the first adjustment parameter is higher than the adjustment precision corresponding to the second adjustment parameter.

[0057] To better understand, the following is a detailed explanation: If the first weight value of pixel A ranks third in the target weight value set and falls within the first range (first to fifth), then the first adjustment parameter pre-configured within the first range is determined as the adjustment parameter for pixel A. If the second weight value of pixel B ranks sixth in the target weight value set and falls within the second range (sixth to tenth), then the second adjustment parameter pre-configured within the second range is determined as the adjustment parameter for pixel B.

[0058] In this implementation, an adjustment parameter can be preset for each ranking in the target weight value set, and the adjustment parameter corresponding to each weight value can be determined to be equal to the adjustment parameter corresponding to the weight value ranking of each weight value in the target weight value set.

[0059] In an optional embodiment, the adjustment parameter corresponding to each weight value in the fifth weight value set is determined according to the preset range in which each weight value in the fifth weight value set is located. This can be determined in the following way: when the first weight value is within a first preset range, the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the first preset range; when the second weight value is within a second preset range, the second adjustment parameter corresponding to the second weight value is determined to be equal to the adjustment parameter corresponding to the second preset range; wherein, the fifth weight value set includes the first weight value and the second weight value, each value in the first preset range is greater than each value in the second preset range, the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment precision in the first adjustment parameter is higher than the adjustment precision in the second adjustment parameter.

[0060] To better understand, the following is a detailed explanation: If there is a first weight value of 5 in the fifth set of weight values, within the first preset range (0-10), then the first adjustment parameter corresponding to this first weight value is determined to be equal to the adjustment parameter corresponding to the first preset range. If there is a second weight value of 15 in the fifth set of weight values, within the first preset range (11-20), then the second adjustment parameter corresponding to this second weight value is determined to be equal to the adjustment parameter corresponding to the second preset range.

[0061] It should be noted that the aforementioned target attention network model and target neural network model are obtained through training. Specifically, this can be achieved in the following way: obtaining a set of sample images; encoding each pixel block in the first set of sample pixel blocks to be encoded in each sample image to obtain a first set of encoded data for each sample image; decoding the first set of encoded data for each sample image to obtain a second set of sample pixel blocks for each sample image; and obtaining a first set of decoded information for each sample image when decoding the first set of encoded data for each sample image.

[0062] Repeat steps one through five until the target loss value between the pixel blocks in the first set of pixel blocks in the current sample image and the pixel blocks in the corresponding third set of pixel blocks is less than or equal to a preset threshold. Then, the attention network model to be trained is determined as the target attention network model, and the neural network model to be trained is determined as the target neural network model.

[0063] Step 1: Select one sample image from the set of sample images as the current sample image;

[0064] Step 2: Input the first set of decoded information of the current sample image into the attention network model to be trained to obtain the set of weight values ​​of the current sample image;

[0065] Step 3: Input the second set of sample pixel blocks and the set of weight values ​​of the current sample image into the neural network model to be trained, and obtain the third set of sample pixel blocks of the current sample image output by the neural network model to be trained;

[0066] Step 4: Determine the target loss value between the pixel blocks in the first set of sample pixel blocks of the current sample image and the pixel blocks in the third set of sample pixel blocks of the current sample image;

[0067] Step 5: If the target loss value between the pixel blocks in the first set of sample pixel blocks of the current sample image and the pixel blocks in the third set of sample pixel blocks of the current sample image is greater than the preset threshold, adjust the parameters in the neural network model to be trained and the attention network model to be trained.

[0068] In one exemplary embodiment, the parameters in the attention network model to be trained are adjusted, including but not limited to the first weight, second weight, third weight, and fourth weight. The parameters in the neural network model to be trained are adjusted, including but not limited to the adjustment parameters corresponding to each range.

[0069] In an optional embodiment, determining the target loss value between pixel blocks in the first set of sample pixel blocks of the current sample image and pixel blocks in the third set of sample pixel blocks of the current sample image can be achieved by: determining a first loss value between pixel blocks in the first set of sample pixel blocks of the current sample image and pixel blocks in the third set of sample pixel blocks of the current sample image using an L2 loss function; determining a second loss value between pixel blocks in the first set of sample pixel blocks of the current sample image and pixel blocks in the third set of sample pixel blocks of the current sample image using an information entropy loss function; and determining the sum of the first loss value and the second loss value as the target loss value.

[0070] In an exemplary embodiment, determining a second loss value between pixel blocks in a first set of sample pixel blocks of the current sample image and pixel blocks in a third set of sample pixel blocks of the current sample image using an information entropy loss function includes: determining a first probability that the pixel value of each pixel in the pixel blocks of the first set of sample pixel blocks is within a third preset range; determining a second probability that the pixel value of each pixel in the pixel blocks of the third set of sample pixel blocks is within the third preset range; and determining the second loss value using the information entropy loss function, the first probability, and the second probability.

[0071] Specifically, the first value of the first probability is determined using the information entropy loss function, and the second value of the second probability is determined using the same information entropy loss function. The absolute value of the difference between the first and second values ​​is then used as the second loss function. The information entropy loss function is as follows:

[0072]

[0073] Where p(x) is the probability value, and 0-N is the third preset range. I(X) is the L2 loss function.

[0074] Obviously, the embodiments described above are merely some embodiments of the present invention, and not all embodiments. To better understand the above image processing method, the process is described below with reference to embodiments, but this is not intended to limit the technical solutions of the embodiments of the present invention. Specifically:

[0075] Figure 3This is a framework diagram (I) of an image processing method according to an embodiment of the present invention. Specifically, the encoded stream of the acquired first image is decoded by a decoder to obtain a compressed image (equivalent to the above-mentioned second pixel block set) and decoding information (equivalent to the above-mentioned combination of first decoding information). The decoding information is then input into a self-attention network model. The compressed image and the output of the self-attention network model (equivalent to the above-mentioned target self-attention network model) are input together into a super-resolution network model (equivalent to the above-mentioned target neural network model) to obtain a second image.

[0076] It should be noted that for the intra-frame coding unit (equivalent to the first pixel block mentioned above) in the compressed image, the pixel block size information and quantization parameter information are input into the self-attention network model to obtain the weight value of each pixel of the corresponding intra-frame coding unit.

[0077] It should be noted that the image processing in this embodiment can also perform super-resolution processing.

[0078] Figure 4 This is a framework diagram (II) of the image processing method according to an embodiment of the present invention. Specifically, the weight value of each pixel in the intra-coding unit output by the self-attention network model can be multiplied by the pixel value of each pixel in the intra-coding unit, and then the result can be input into the super-resolution network model.

[0079] Figure 5 This is a framework diagram (III) of the image processing method according to an embodiment of the present invention. It should be noted that, for the inter-frame coding unit (equivalent to the second pixel block mentioned above) in the compressed image, the quantization parameter information and the residual information of the pixel are input into the self-attention network model to obtain the weight value of each pixel of the corresponding inter-frame coding unit.

[0080] Figure 6 This is a framework diagram (four) of the image processing method according to an embodiment of the present invention. Specifically, the weight value of each pixel in the inter-frame coding unit output by the self-attention network model can be multiplied by the pixel value of each pixel in the inter-frame coding unit, and then the result can be input into the super-resolution network model.

[0081] To better understand, the following is a detailed explanation, which can be divided into the following steps:

[0082] Step 1: Input the current bitstream, decode it, and output video data (equivalent to the second pixel block set mentioned above) and decoding information. If it is an intra-frame coding unit (equivalent to the first pixel block mentioned above), select cu-size / cu-qp from the decoding information and input it into the attention network model, such as... Figure 3If it is an inter-frame coding unit (equivalent to the second pixel block mentioned above), select cu-res / cu-qp information to input into the attention network model, such as... Figure 5 It should be noted that extracting different bitstream information based on different encoding methods is one of the inventive aspects of this invention.

[0083] Step 2: Input the video image data information and the information output by att-net (self-attention network model) into sr-net (super-resolution network model).

[0084] Figure 3 and Figure 4 For intra-frame coding units, cu-size / cu-qp information is input into the Att-Net. For pixels at different locations, a smaller cu-size indicates richer texture information, meaning pixels in these regions are more important and require more attention. Similarly, for pixels at different locations, a smaller cu-qp is considered to indicate richer texture information, meaning pixels in these regions are more important and require more attention. Figure 4 The output of att-net is multiplied by data, which is the simplest spatial attention mechanism, and then input into sr-net.

[0085] Figure 5 and Figure 6 For inter-frame coding units, cu-res / cu-qp information is input into the att-net. For pixels at different locations, a denser cu-res indicates richer added texture information, meaning pixels in these areas are more important and require more attention. Conversely, a smaller cu-qp indicates richer texture information, again meaning pixels in these areas are more important and require more attention. Figure 6 The output of att-net is multiplied by data, which is the simplest spatial attention mechanism, and then input into sr-net.

[0086] The second inventive point of this invention is how to use the bitstream decoding information to dynamically improve the data information and thus improve the SR-Net effect.

[0087] Step 3: SR-Net outputs a video image (equivalent to the second image mentioned above). The loss is calculated using two methods: the traditional L2 loss and the information entropy loss between the input and output video images. Information entropy ensures that the output image and the current image have consistent basic features, preventing large fluctuations in spatial information; it also ensures that the output image and the reference image have consistent basic features, preventing large fluctuations in temporal information. The overall loss is as follows:

[0088] Loss = λ1 * Loss SR +λ2*Loss in

[0089] The weight coefficient λ can be trained or has a custom size. The weight for each loss is calculated as follows.

[0090]

[0091] Among them, LOSS SR Let t be the L2 loss, h be the height, and w be the length.

[0092]

[0093] Among them, LOSS in The information entropy loss is represented by gt, which denotes the ground truth image, and H function, which represents the information entropy. Specifically, the histograms of SR and gt are calculated first. Then, the probability p(x) of a pixel value x falling within the range of 0 to N is obtained. Finally, the information entropy of the image is calculated by subtracting the gt and SR images and squaring the result, yielding the mean square error of the information entropy. The formula is as follows:

[0094]

[0095] The multiple loss calculation methods are the third inventive point of this invention.

[0096] The above steps then complete one training iteration of the self-attention network model and the super-resolution network model.

[0097] It should be noted that, Figure 3 The input bitstream can be either image or video. The decoder standard can be either an image decoding protocol or a video decoding protocol, including the H.26x series, MPEG series, AVS series, and VPx / AV1 series.

[0098] Figure 3 The combination of att-net and sr-net can be achieved in the following ways:

[0099] Firstly, the attention mechanism can be channel attention, spatial attention, or a combination of multiple attention mechanisms.

[0100] Secondly, the attention output to SR-Net can be at the start of the input, the middle of the input, or the end of the input.

[0101] Super-resolution (SR) is a process that improves the resolution of an original image through hardware or software methods. The process of obtaining a high-resolution image from a series of low-resolution images is called super-resolution reconstruction. The core idea of ​​super-resolution reconstruction is to trade temporal bandwidth (acquiring one or more frames of the same scene) for spatial resolution, thus achieving the conversion from temporal resolution to spatial resolution.

[0102] Furthermore, the improvements in decoding information made in this invention can also be applied to video image super-resolution / super-resolution (SR).

[0103] In practice, Super-Resolution (SR) technology can be divided into two categories: video image restoration and video image interpolation. Video image interpolation includes video image resolution changes and video image frame rate changes. Video image resolution changes can be infinitely magnified or reduced, while video image frame rate changes can be frame interpolation or frame extraction. Generally speaking, video super-resolution originates from image super-resolution, and its purpose is to recover the target image from one or more reference images. The difference between them is also obvious. Since video is composed of multiple frames, traditional methods usually utilize inter-frame and intra-frame information for restoration.

[0104] Compared to traditional SR, deep learning and neural networks can greatly improve the subjective and objective quality of video images. These include SRCNN, DRCN, ESPCN, VESPCN, and SRGAN, which are significantly higher than the video image quality obtained by traditional SR methods. Therefore, combining deep learning and neural networks in SR methods is the future development trend.

[0105] Traditional methods often employ deep neural network-based approaches, which only utilize information from the video image itself and do not take advantage of information that may be provided during the decoding process.

[0106] Right now Figure 3 The following are alternative solutions for the output of the SR-Net component:

[0107] The super-resolution domain for video images includes resolution scaling, frame interpolation, and enhancement. Therefore, the SR post-processing module functions in three ways: 1. Resolution scaling, adding an image upsampling component; 2. Frame interpolation, adding an image interpolated component; 3. Enhancement, adding an image restoration component. All of these are included in SR-Net.

[0108] This invention combines decoded information and deep neural networks. By using a self-attention network, the decoded information is added to the deep neural network to improve the SR effect. This not only greatly improves training efficiency but also effectively reduces the complexity of the deep neural network.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0110] This embodiment also provides an image processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0111] Figure 7 This is a frame diagram (a) of an image processing apparatus according to an embodiment of the present invention, the apparatus comprising:

[0112] The first acquisition module 71 is used to acquire a set of first pixel blocks to be encoded in the first image, and to encode each pixel block in the first pixel block set to obtain a first encoded data set, wherein each pixel block corresponds to an encoding method;

[0113] The decoding module 72 is used to decode the first encoded data set to obtain a second pixel block set, and to obtain a first decoding information set when decoding the first encoded data set, wherein the decoding information in the first decoding information set corresponds one-to-one with each pixel block in the second pixel block set;

[0114] The second acquisition module 73 is used to input the first decoded information set into the target attention network model to obtain a target weight value set, wherein the target weight value set includes a weight value that corresponds one-to-one with each pixel in the second pixel block set, and the weight value corresponding to each pixel is used to represent the importance of each pixel in the second pixel block set;

[0115] The third acquisition module 74 is used to input the second pixel block set and the target weight value set into the target neural network model to obtain the third pixel block set output by the target neural network model. The target neural network model is used to adjust the pixel values ​​of the pixels in the second pixel block set according to the target weight value set to obtain the third pixel block set, and the third pixel block set is used to form the second image.

[0116] Through the above modules, during the transmission of the first image, each pixel block in the first pixel block set to be encoded in the first image is encoded to obtain a first encoded data set. Then, at the receiving end, the obtained first encoded data set is decoded to obtain a second pixel block set, and a first decoding information set is obtained when decoding the first encoded data set. This first decoding information set is input into a target attention network model to obtain a target weight value set. Then, the second pixel block set and the target weight value set are input into a target neural network model to obtain a third pixel block set output by the target neural network model. The third pixel block set is used to form the second image. Using this technical solution, for each pixel block in the image, the target neural network model can determine the importance of pixels in the pixel block based on the decoding information, and thus focus on processing important pixels, improving the image processing effect. This solves the problem that traditional image processing based on neural network algorithms only utilizes the image data obtained after decoding the encoded stream, without utilizing the decoding information obtained after decoding the encoded stream, resulting in poor image processing performance.

[0117] Optionally, the decoding module 72 is further configured to, when the encoding method corresponding to the first pixel block is intra-frame coding, acquire first decoding information when decoding the first encoded data, wherein the first pixel block set includes the first pixel block, the first encoded data is encoded data obtained by encoding the first pixel block using the intra-frame coding method, and the first decoding information includes the size information of the first pixel block and a first quantization parameter, wherein the first quantization parameter is the quantization parameter used to encode the first pixel block; and / or, when the encoding method corresponding to the second pixel block is inter-frame coding, acquire second decoding information when decoding the second encoded data, wherein the first pixel block set includes the second pixel block, the second encoded data is encoded data obtained by encoding the second pixel block using the inter-frame coding method, and the second decoding information includes a residual information set and a second quantization parameter, wherein the second quantization parameter is the quantization parameter used to encode the second pixel block, and the residual information set includes residual information corresponding to each pixel in the second pixel block.

[0118] Optionally, the second acquisition module 73 is further configured to input the first decoding information into the target attention network model to obtain a first weight value set when the encoding method corresponding to the first pixel block is the intra-frame encoding method, wherein the first weight value set includes weight values ​​corresponding one-to-one with each pixel in the third pixel block, the target weight value set includes the first weight set, the second pixel block set includes the third pixel block, the third pixel block is a pixel block obtained by decoding the first encoded data of the first pixel block, and the target attention model is configured to determine the first weight set of the third pixel block based on the product of the size information of the first pixel block and the first weight, and the product of the first quantization parameter and the second weight.

[0119] Optionally, the second acquisition module 73 is further configured to input the second decoding information into the target attention network model to obtain a second weight value set when the encoding method corresponding to the second pixel block is the inter-frame encoding method, wherein the second weight value set includes weight values ​​that correspond one-to-one with each pixel in the fourth pixel block, the target weight value set includes the second weight set, the second pixel block set includes the fourth pixel block, the fourth pixel block is a pixel block obtained after decoding the second encoded data of the second pixel block, and the target attention model is configured to determine the second weight set of the fourth pixel block based on the product of the second quantization parameter and the third weight, and the product of the residual information set and the fourth weight.

[0120] Optionally, the third acquisition module 74 is further configured to input the fifth pixel block in the second pixel block set and the fifth weight value set corresponding to the fifth pixel block in the target weight value set into the target neural network model to obtain the sixth pixel block in the third pixel block set. The target neural network model is configured to adjust the pixel values ​​of the pixels in the fifth pixel block according to the adjustment parameters corresponding to the weight values ​​in the fifth weight value set to obtain the sixth pixel block. The adjustment parameters corresponding to the fifth weight value set include the allowable adjustment range and / or the allowable adjustment precision of the pixel values ​​when adjusting the pixel values ​​of the pixels in the fifth pixel block.

[0121] Figure 8 This is a framework diagram (II) of an image processing apparatus according to an embodiment of the present invention. The apparatus includes: a determination module 75 and a training module 76.

[0122] Optionally, the determining module 75 is further configured to determine the adjustment parameter corresponding to each weight value in the fifth weight value set by one of the following: determining the adjustment parameter corresponding to each weight value in the fifth weight value set according to the weight value ranking of each weight value in the fifth weight value set; determining the adjustment parameter corresponding to each weight value in the fifth weight value set according to the weight value ranking of each weight value in the target weight value set; or determining the adjustment parameter corresponding to each weight value in the fifth weight value set according to the preset range in which each weight value in the fifth weight value set is located.

[0123] Optionally, the determining module 75 is further configured to, when the ranking of the first weight value in the fifth set of weight values ​​is within a first range, determine the first adjustment parameter corresponding to the first weight value to be equal to the adjustment parameter corresponding to the first range; and when the ranking of the second weight value in the fifth set of weight values ​​is within a second range, determine the second adjustment parameter corresponding to the first weight value to be equal to the adjustment parameter corresponding to the second range; wherein, each of the multiple ranges including the first range and the second range is pre-configured with a corresponding adjustment parameter, the fifth set of weight values ​​includes the first weight value and the second weight value, each ranking in the first range is higher than each ranking in the second range, the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter. The adjustment precision is higher than the adjustment precision corresponding to the second adjustment parameter; or, if the weight value ranking of the first weight value in the fifth weight value set is higher than the weight value ranking of the second weight value in the fifth weight value set, the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the weight value ranking of the first weight value in the fifth weight value set, and the second adjustment parameter corresponding to the second weight value is determined to be equal to the adjustment parameter corresponding to the weight value ranking of the second weight value in the fifth weight value set, wherein each weight value ranking in the fifth weight value set is pre-configured with a corresponding adjustment parameter, the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment precision in the first adjustment parameter is higher than the adjustment precision corresponding to the second adjustment parameter.

[0124] Optionally, the determining module 75 is further configured to, when the ranking of the first weight value in the target weight value set is within a first range, determine the first adjustment parameter corresponding to the first weight value to be equal to the adjustment parameter corresponding to the first range; and when the ranking of the second weight value in the target weight value set is within a second range, determine the second adjustment parameter corresponding to the first weight value to be equal to the adjustment parameter corresponding to the second range; wherein, each of the multiple ranges including the first range and the second range is pre-configured with a corresponding adjustment parameter, the fifth weight value set includes the first weight value and the second weight value, each ranking in the first range is higher than each ranking in the second range, the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter. The adjustment precision is higher than the adjustment precision corresponding to the second adjustment parameter; or, if the weight value ranking of the first weight value in the target weight value set is higher than the weight value ranking of the second weight value in the target weight value set, the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the weight value ranking of the first weight value in the target weight value set, and the second adjustment parameter corresponding to the second weight value is determined to be equal to the adjustment parameter corresponding to the weight value ranking of the second weight value in the target weight value set, wherein each weight value ranking in the target weight value set is pre-configured with a corresponding adjustment parameter, the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment precision in the first adjustment parameter is higher than the adjustment precision corresponding to the second adjustment parameter.

[0125] Optionally, the determining module 75 is further configured to, when the first weight value is within a first preset range, determine the first adjustment parameter corresponding to the first weight value as equal to the adjustment parameter corresponding to the first preset range; and when the second weight value is within a second preset range, determine the second adjustment parameter corresponding to the second weight value as equal to the adjustment parameter corresponding to the second preset range; wherein, the fifth weight value set includes the first weight value and the second weight value, each value in the first preset range is greater than each value in the second preset range, the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment precision in the first adjustment parameter is higher than the adjustment precision in the second adjustment parameter.

[0126] Optionally, the training module 76 is further configured to acquire a set of sample images, encode each pixel block in the first set of sample pixel blocks to be encoded in each sample image in the set of sample images to obtain a first set of encoded data for each sample image; decode the first set of encoded data for each sample image to obtain a second set of sample pixel blocks for each sample image, and acquire a first set of decoded information for each sample image when decoding the first set of encoded data for each sample image; repeat the following steps until the target loss value between the pixel blocks in the first set of sample pixel blocks of the current sample image and the pixel blocks in the corresponding third set of sample pixel blocks is less than or equal to a preset threshold, determine the attention network model to be trained as the target attention network model, and determine the neural network model to be trained as the target neural network model: select a sample from the set of sample images. This image is used as the current sample image; the first set of decoded information of the current sample image is input into the attention network model to be trained to obtain the weight value set of the current sample image; the second set of sample pixel blocks and the weight value set of the current sample image are input into the neural network model to be trained to obtain the third set of sample pixel blocks of the current sample image output by the neural network model to be trained; the target loss value between the pixel blocks in the first set of sample pixel blocks of the current sample image and the pixel blocks in the third set of sample pixel blocks of the current sample image is determined; if the target loss value between the pixel blocks in the first set of sample pixel blocks of the current sample image and the pixel blocks in the third set of sample pixel blocks of the current sample image is greater than the preset threshold, the parameters in the neural network model to be trained and the attention network model to be trained are adjusted.

[0127] Optionally, the training module 76 is further configured to determine a first loss value between pixel blocks in the first set of sample pixel blocks of the current sample image and pixel blocks in the third set of sample pixel blocks of the current sample image using an L2 loss function; determine a second loss value between pixel blocks in the first set of sample pixel blocks of the current sample image and pixel blocks in the third set of sample pixel blocks of the current sample image using an information entropy loss function; and determine the sum of the first loss value and the second loss value as the target loss value.

[0128] Optionally, the training module 76 is further configured to determine a first probability that the pixel value of each pixel in the pixel block of the first sample pixel block set is within a third preset range; determine a second probability that the pixel value of each pixel in the pixel block of the third sample pixel block set is within the third preset range; and determine a second loss value using the information entropy loss function, the first probability, and the second probability.

[0129] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0130] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0131] S1, obtain a set of first pixel blocks to be encoded in the first image, encode each pixel block in the first pixel block set to obtain a first encoded data set, wherein each pixel block corresponds to an encoding method;

[0132] S2, decode the first encoded data set to obtain the second pixel block set, and obtain the first decoding information set when decoding the first encoded data set, wherein the decoding information in the first decoding information set corresponds one-to-one with each pixel block in the second pixel block set;

[0133] S3, input the first decoded information set into the target attention network model to obtain the target weight value set, wherein the target weight value set includes a weight value that corresponds one-to-one with each pixel in the second pixel block set, and the weight value corresponding to each pixel is used to represent the importance of each pixel in the second pixel block set;

[0134] S4, the second pixel block set and the target weight value set are input into the target neural network model to obtain the third pixel block set output by the target neural network model. The target neural network model is used to adjust the pixel values ​​of the pixels in the second pixel block set according to the target weight value set to obtain the third pixel block set. The third pixel block set is used to form the second image.

[0135] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0136] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0137] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0138] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0139] S1, obtain a set of first pixel blocks to be encoded in the first image, encode each pixel block in the first pixel block set to obtain a first encoded data set, wherein each pixel block corresponds to an encoding method;

[0140] S2, decode the first encoded data set to obtain the second pixel block set, and obtain the first decoding information set when decoding the first encoded data set, wherein the decoding information in the first decoding information set corresponds one-to-one with each pixel block in the second pixel block set;

[0141] S3, input the first decoded information set into the target attention network model to obtain the target weight value set, wherein the target weight value set includes a weight value that corresponds one-to-one with each pixel in the second pixel block set, and the weight value corresponding to each pixel is used to represent the importance of each pixel in the second pixel block set;

[0142] S4, the second pixel block set and the target weight value set are input into the target neural network model to obtain the third pixel block set output by the target neural network model. The target neural network model is used to adjust the pixel values ​​of the pixels in the second pixel block set according to the target weight value set to obtain the third pixel block set. The third pixel block set is used to form the second image.

[0143] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0144] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0145] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0146] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: Obtain a set of first pixel blocks to be encoded in a first image, encode each pixel block in the first pixel block set to obtain a first encoded data set, wherein each pixel block corresponds to an encoding method; The first encoded data set is decoded to obtain the second pixel block set, and the first decoding information set is obtained when the first encoded data set is decoded, wherein the decoding information in the first decoding information set corresponds one-to-one with each pixel block in the second pixel block set; The first set of decoded information is input into the target attention network model to obtain a set of target weight values. The set of target weight values ​​includes a weight value that corresponds one-to-one with each pixel in the second set of pixel blocks. The weight value corresponding to each pixel is used to represent the importance of each pixel in the second set of pixel blocks. The second pixel block set and the target weight value set are input into the target neural network model to obtain the third pixel block set output by the target neural network model. The target neural network model is used to adjust the pixel values ​​of the pixels in the second pixel block set according to the target weight value set to obtain the third pixel block set, which is used to form the second image.

2. The method according to claim 1, characterized in that, The step of obtaining the first decoding information set when decoding the first encoded data set includes: When the encoding method corresponding to the first pixel block is intra-frame coding, first decoding information is obtained when decoding the first encoded data. The first pixel block set includes the first pixel block, the first encoded data is encoded data obtained by encoding the first pixel block using the intra-frame coding method, and the first decoding information includes the size information of the first pixel block and a first quantization parameter, wherein the first quantization parameter is the quantization parameter used to encode the first pixel block; and / or When the encoding method corresponding to the second pixel block is inter-frame coding, second decoding information is obtained when decoding the second encoded data. The first pixel block set includes the second pixel block, the second encoded data is the encoded data obtained by encoding the second pixel block using the inter-frame coding method, and the second decoding information includes a residual information set and a second quantization parameter. The second quantization parameter is the quantization parameter used to encode the second pixel block, and the residual information set includes the residual information corresponding to each pixel in the second pixel block.

3. The method according to claim 2, characterized in that, The step of inputting the first decoded information set into the target attention network model to obtain the target weight value set includes: When the encoding method corresponding to the first pixel block is the intra-frame encoding method, the first decoded information is input into the target attention network model to obtain a first weight value set. The first weight value set includes weight values ​​that correspond one-to-one with each pixel in the third pixel block. The target weight value set includes the first weight set. The second pixel block set includes the third pixel block. The third pixel block is a pixel block obtained by decoding the first encoded data of the first pixel block. The target attention network model is used to determine the first weight set of the third pixel block based on the product of the size information of the first pixel block and the first weight, and the product of the first quantization parameter and the second weight.

4. The method according to claim 2, characterized in that, The step of inputting the first decoded information set into the target attention network model to obtain the target weight value set includes: When the encoding method corresponding to the second pixel block is the inter-frame encoding method, the second decoded information is input into the target attention network model to obtain a second weight value set. The second weight value set includes weight values ​​that correspond one-to-one with each pixel in the fourth pixel block. The target weight value set includes the second weight set. The second pixel block set includes the fourth pixel block. The fourth pixel block is a pixel block obtained by decoding the second encoded data of the second pixel block. The target attention network model is used to determine the second weight set of the fourth pixel block based on the product of the second quantization parameter and the third weight, and the product of the residual information set and the fourth weight.

5. The method according to claim 1, characterized in that, The step of inputting the second pixel block set and the target weight value set into the target neural network model to obtain the third pixel block set output by the target neural network model includes: The fifth pixel block in the second pixel block set and the fifth weight value set corresponding to the fifth pixel block in the target weight value set are input into the target neural network model to obtain the sixth pixel block in the third pixel block set. The target neural network model is used to adjust the pixel values ​​of the pixels in the fifth pixel block according to the adjustment parameters corresponding to the weight values ​​in the fifth weight value set to obtain the sixth pixel block. The adjustment parameters corresponding to the fifth weight value set include the allowable adjustment range and / or the allowable adjustment precision of the pixel values ​​when adjusting the pixel values ​​of the pixels in the fifth pixel block.

6. The method according to claim 5, characterized in that, The method further includes: The adjustment parameter for each weight value in the fifth set of weight values ​​is determined by one of the following: The adjustment parameter corresponding to each weight value in the fifth weight value set is determined based on the ranking of each weight value in the fifth weight value set. The adjustment parameter corresponding to each weight value in the fifth weight value set is determined based on the ranking of each weight value in the target weight value set. The adjustment parameter corresponding to each weight value in the fifth set of weight values ​​is determined based on the preset range in which each weight value in the fifth set of weight values ​​is located.

7. The method according to claim 6, characterized in that, The adjustment parameter corresponding to each weight value in the fifth weight value set is determined based on the ranking of each weight value in the fifth weight value set, including: When the ranking of the first weight value in the fifth set of weight values ​​is within a first range, the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the first range; when the ranking of the second weight value in the fifth set of weight values ​​is within a second range, the second adjustment parameter corresponding to the second weight value is determined to be equal to the adjustment parameter corresponding to the second range; wherein, each of the multiple ranges including the first range and the second range is pre-configured with a corresponding adjustment parameter, the fifth set of weight values ​​includes the first weight value and the second weight value, each ranking in the first range is higher than each ranking in the second range, the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment precision in the first adjustment parameter is higher than the adjustment precision corresponding to the second adjustment parameter; or If the ranking of the first weight value in the fifth set of weight values ​​is higher than that of the second weight value in the fifth set of weight values, then the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the ranking of the first weight value in the fifth set of weight values, and the second adjustment parameter corresponding to the second weight value is determined to be equal to the adjustment parameter corresponding to the ranking of the second weight value in the fifth set of weight values. Each weight value ranking in the fifth set of weight values ​​is pre-configured with a corresponding adjustment parameter, and the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment precision in the first adjustment parameter is higher than the adjustment precision corresponding to the second adjustment parameter.

8. The method according to claim 6, characterized in that, The adjustment parameter corresponding to each weight value in the fifth weight value set is determined based on the ranking of each weight value in the target weight value set, including: When the ranking of the first weight value in the target weight value set is within a first range, the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the first range; when the ranking of the second weight value in the target weight value set is within a second range, the second adjustment parameter corresponding to the second weight value is determined to be equal to the adjustment parameter corresponding to the second range; wherein, each of the multiple ranges including the first range and the second range is pre-configured with a corresponding adjustment parameter, the fifth weight value set includes the first weight value and the second weight value, each ranking in the first range is higher than each ranking in the second range, the adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment precision in the first adjustment parameter is higher than the adjustment precision corresponding to the second adjustment parameter; or If the ranking of the first weight value in the target weight value set is higher than that of the second weight value in the target weight value set, then the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the ranking of the first weight value in the target weight value set, and the second adjustment parameter corresponding to the second weight value is determined to be equal to the adjustment parameter corresponding to the ranking of the second weight value in the target weight value set. Each weight value ranking in the target weight value set is pre-configured with a corresponding adjustment parameter, and the adjustment range of the first adjustment parameter is greater than the adjustment range of the second adjustment parameter and / or the adjustment precision of the first adjustment parameter is higher than the adjustment precision corresponding to the second adjustment parameter.

9. The method according to claim 6, characterized in that, The adjustment parameters corresponding to each weight value in the fifth set of weight values ​​are determined based on the preset range in which each weight value falls, including: When the first weight value is within a first preset range, the first adjustment parameter corresponding to the first weight value is determined to be equal to the adjustment parameter corresponding to the first preset range; When the second weight value is within the second preset range, the second adjustment parameter corresponding to the second weight value is determined to be equal to the adjustment parameter corresponding to the second preset range; The fifth set of weight values ​​includes the first weight value and the second weight value. Each value in the first preset range is greater than each value in the second preset range. The adjustment range in the first adjustment parameter is greater than the adjustment range in the second adjustment parameter and / or the adjustment precision in the first adjustment parameter is higher than the adjustment precision in the second adjustment parameter.

10. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain a set of sample images, and encode each pixel block in the first set of sample pixel blocks to be encoded in each sample image in the set of sample images to obtain the first encoded data set of each sample image; Decode the first encoded data set of each sample image to obtain the second sample pixel block set of each sample image, and obtain the first decoded information set of each sample image when decoding the first encoded data set of each sample image; Repeat the following steps until the target loss value between the pixel blocks in the first set of pixel blocks of the current sample image and the pixel blocks in the corresponding third set of pixel blocks is less than or equal to a preset threshold. Then, the attention network model to be trained is determined as the target attention network model, and the neural network model to be trained is determined as the target neural network model. Select one sample image from the set of sample images as the current sample image; The first set of decoded information of the current sample image is input into the attention network model to be trained to obtain the set of weight values ​​of the current sample image; The second set of sample pixel blocks and the set of weight values ​​of the current sample image are input into the neural network model to be trained to obtain the third set of sample pixel blocks of the current sample image output by the neural network model to be trained. Determine the target loss value between pixel blocks in the first set of sample pixel blocks of the current sample image and pixel blocks in the third set of sample pixel blocks of the current sample image; If the target loss value between a pixel block in the first set of pixel blocks of the current sample image and a pixel block in the third set of pixel blocks of the current sample image is greater than the preset threshold, the parameters in the neural network model to be trained and the attention network model to be trained are adjusted.

11. The method according to claim 10, characterized in that, Determining the target loss value between pixel blocks in the first set of sample pixel blocks of the current sample image and pixel blocks in the third set of sample pixel blocks of the current sample image includes: The first loss value between the pixel blocks in the first set of sample pixel blocks of the current sample image and the pixel blocks in the third set of sample pixel blocks of the current sample image is determined by the L2 loss function. The second loss value between the pixel blocks in the first set of sample pixel blocks of the current sample image and the pixel blocks in the third set of sample pixel blocks of the current sample image is determined by the information entropy loss function; The sum of the first loss value and the second loss value is determined as the target loss value.

12. The method according to claim 11, characterized in that, The second loss value is determined by using the information entropy loss function to compare pixel blocks in the first set of sample pixel blocks of the current sample image with pixel blocks in the third set of sample pixel blocks of the current sample image, including: Determine the first probability that the pixel value of each pixel in the pixel block of the first sample pixel block set is within a third preset range; Determine the second probability that the pixel value of each pixel in the pixel block of the third sample pixel block set is within the third preset range; The second loss value is determined using the information entropy loss function, the first probability, and the second probability.

13. An image processing apparatus, characterized in that, include: The first acquisition module is used to acquire a set of first pixel blocks to be encoded in a first image, and to encode each pixel block in the first pixel block set to obtain a first encoded data set, wherein each pixel block corresponds to an encoding method; The decoding module is used to decode the first encoded data set to obtain a second pixel block set, and to obtain a first decoding information set when decoding the first encoded data set, wherein the decoding information in the first decoding information set corresponds one-to-one with each pixel block in the second pixel block set; The second acquisition module is used to input the first decoded information set into the target attention network model to obtain a target weight value set, wherein the target weight value set includes a weight value that corresponds one-to-one with each pixel in the second pixel block set, and the weight value corresponding to each pixel is used to represent the importance of each pixel in the second pixel block set; The third acquisition module is used to input the second pixel block set and the target weight value set into the target neural network model to obtain the third pixel block set output by the target neural network model. The target neural network model is used to adjust the pixel values ​​of the pixels in the second pixel block set according to the target weight value set to obtain the third pixel block set, and the third pixel block set is used to form the second image.

14. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 12 when it is run.

15. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to perform the method described in any one of claims 1 to 12 via the computer program.

Images