Image processing method and device, equipment and storage medium

Abstract

The invention provides an image processing method and device, equipment and a storage medium, and can be applied to the technical field of image processing. The method comprises the following steps: when facing a plurality of similar images, encoding and decoding an original image through an encoding end and a decoding end to obtain a first decoded image; performing transformation processing on the reference image through the decoding end to obtain an auxiliary information image; performing fusion processing on the auxiliary information image and the first decoded image to obtain a fused image; determining a reconstructed image corresponding to the original image based on the fused image; according to the method, only the original image needs to be coded, so that the calculation amount of a coding end is reduced, the coding efficiency is improved, meanwhile, the image can be reconstructed based on the auxiliary information image and the decoded image, the fuzzy degree of the reconstructed image is reduced, and the image quality of the decoded image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] During driving, the vehicle's driving recorder will record the images of the vehicle while driving. In order to save storage space and improve usage efficiency, the data generated by the driving recorder is usually compressed. Existing solutions for compressing driving record data usually use H.264 video compression solutions and JPEG static image compression solutions.

[0003] However, the JPEG static image compression scheme is mainly used for compressing single-frame images. Although it provides a high compression ratio, the compressed image will have obvious image quality loss; although the H.264 video compression scheme can improve the image quality of compressed images, it needs to encode and decode the compressed images one by one, which increases the computational complexity of the codec and affects the coding efficiency.

[0004] Therefore, how to improve the image quality of compressed images while reducing the consumption of the encoding end and improving the encoding efficiency is the problem to be solved at present. Summary of the invention

[0005] The present application provides an image processing method, apparatus, device and storage medium to solve the problem in the prior art of how to improve the image quality of compressed images while reducing the consumption of the encoding end and improving the encoding efficiency.

[0006] In a first aspect, the present application provides an image processing method, which is applied to an image processing device, the image processing device comprising: an encoding end and a decoding end, the method comprising:

[0007] The encoding and decoding end perform encoding and decoding processing on the original image to obtain a first decoded image;

[0008] The reference image is transformed by the decoding end to obtain an auxiliary information image;

[0009] fusing the auxiliary information image and the first decoded image to obtain a fused image;

[0010] Based on the fused image, a reconstructed image corresponding to the original image is determined.

[0011] In a possible implementation, the decoding end includes: a decoding module and a transformation module, and the decoding end transforms the reference image to obtain the auxiliary information image, including:

[0012] Decoding the reference image through the decoding module to obtain a second decoded image;

[0013] A target image block set corresponding to a plurality of image blocks in the first decoded image is determined through the transformation module, and the target image block set is determined as the auxiliary information image, wherein the target image block set is determined from the second decoded image.

[0014] In a possible implementation manner, determining a target image block set corresponding to a plurality of image blocks in the first decoded image includes:

[0015] respectively performing block processing on the first decoded image and the second decoded image to obtain a plurality of first image blocks corresponding to the first decoded image and a plurality of second image blocks corresponding to the second decoded image;

[0016] For any one of the plurality of first image blocks, determining a correlation between the first image block and the plurality of second image blocks;

[0017] Using the image block with the highest correlation among the multiple second image blocks as the target image block corresponding to the first image block;

[0018] A plurality of target image blocks corresponding to the plurality of first image blocks are taken as the target image block set.

[0019] In a possible implementation, the decoding end further includes: a multi-scale fusion module, wherein the fusion processing of the auxiliary information image and the first decoded image to obtain a fused image includes:

[0020] Using the multi-scale fusion module, the auxiliary information image and the first decoded image are expanded according to a plurality of preset resolutions to obtain a first sub-fusion image and a second sub-fusion image corresponding to each resolution;

[0021] For any one of the multiple resolutions, fusing the first sub-fusion image and the second sub-fusion image corresponding to the resolution to obtain a third sub-fusion image corresponding to the resolution;

[0022] An image pyramid fusion algorithm is used to perform fusion processing on a plurality of third sub-fusion images to obtain the fusion image.

[0023] In a possible implementation, the decoding end further includes: an image reconstruction module, and the determining, based on the fused image, a reconstructed image corresponding to the original image includes:

[0024] Inputting the fused image into the image reconstruction module to obtain a reconstruction result output by the image reconstruction module, wherein the reconstruction result includes: a reconstructed image corresponding to the original image;

[0025] Wherein, the image reconstruction module is trained based on historical original images and corresponding historical reconstructed images.

[0026] In a possible implementation manner, the reconstruction result further includes: a distortion weight, and the method further includes:

[0027] Determining whether the distortion weight reaches a preset weight;

[0028] When the distortion weight does not reach a preset weight, the encoding parameters of the encoding end are updated based on the distortion weight.

[0029] In a second aspect, the present application provides an image processing apparatus, which is applied to an image processing device, wherein the image processing device comprises: an encoding end and a decoding end, and the apparatus comprises:

[0030] The processing module is used to perform encoding and decoding processing on the original image through the encoding end and the decoding end to obtain a first decoded image; perform transformation processing on the reference image through the decoding end to obtain an auxiliary information image; perform fusion processing on the auxiliary information image and the first decoded image to obtain a fused image; and determine a reconstructed image corresponding to the original image based on the fused image.

[0031] In a third aspect, the present application provides an image processing device, including: an encoding end and a decoding end;

[0032] The encoding end is used to encode the original image;

[0033] The decoding end is used to decode the original image after the encoding process to obtain a first decoded image;

[0034] The decoding end is further used to transform the reference image to obtain an auxiliary information image; fuse the auxiliary information image and the first decoded image to obtain a fused image; and determine a reconstructed image corresponding to the original image based on the fused image.

[0035] In a possible implementation, the decoding end includes: a decoding module and a transformation module;

[0036] The decoding module is used to decode the reference image to obtain a second decoded image;

[0037] The transformation module is used to determine a target image block set corresponding to multiple image blocks in the first decoded image, and determine the target image block set as the auxiliary information image, wherein the target image block set is determined from the second decoded image.

[0038] In a possible implementation, the transform module is used to perform block processing on the first decoded image and the second decoded image respectively to obtain a plurality of first image blocks corresponding to the first decoded image and a plurality of second image blocks corresponding to the second decoded image;

[0039] For any one of the plurality of first image blocks, determining a correlation between the first image block and the plurality of second image blocks;

[0040] Using the image block with the highest correlation among the multiple second image blocks as the target image block corresponding to the first image block;

[0041] A plurality of target image blocks corresponding to the plurality of first image blocks are taken as the target image block set.

[0042] In a possible implementation, the decoding end further includes: a multi-scale fusion module;

[0043] The multi-scale fusion module is used to perform expansion processing on the auxiliary information image and the first decoded image according to a plurality of preset resolutions to obtain a first sub-fusion image and a second sub-fusion image corresponding to each resolution;

[0044] For any one of the multiple resolutions, fusing the first sub-fusion image and the second sub-fusion image corresponding to the resolution to obtain a third sub-fusion image corresponding to the resolution;

[0045] An image pyramid fusion algorithm is used to perform fusion processing on a plurality of third sub-fusion images to obtain the fusion image.

[0046] In a possible implementation, the decoding end further includes: an image reconstruction module;

[0047] The image reconstruction module is used to perform image reconstruction processing on the input fused image and output a reconstruction result, wherein the reconstruction result includes: a reconstructed image corresponding to the original image;

[0048] Wherein, the image reconstruction module is trained based on historical original images and corresponding historical reconstructed images.

[0049] In a possible implementation, the image processing device further includes: an updating module;

[0050] The updating module is used to determine whether the distortion weight reaches a preset weight;

[0051] When the distortion weight does not reach a preset weight, the encoding parameters of the encoding end are updated based on the distortion weight.

[0052] In a fourth aspect, the present application provides an image processing device, including: a memory, a processor, and a communication interface;

[0053] Wherein, the memory stores computer-executable instructions, and the communication interface is used to communicate with other devices via a transmission medium;

[0054] The processor executes the computer-executable instructions stored in the memory to implement the image processing method as described in the first aspect and various possible implementations of the first aspect.

[0055] In a fifth aspect, the present application provides a computer storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the image processing method as described in the first aspect and various possible implementations of the first aspect.

[0056] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the image processing method as described in the first aspect and various possible implementation methods of the first aspect.

[0057] The present application provides an image processing method, device, equipment and storage medium. When a detection device determines that there is an obstacle in a blind spot, the collision result of the vehicle and the obstacle is determined, that is, whether there is a collision risk between the vehicle and the obstacle. If so, the risk assessment index of the vehicle is determined, and the risk assessment of the vehicle is performed based on the risk assessment index to obtain the risk level of the vehicle. The method uses a preset risk assessment index to perform a risk assessment on the vehicle, thereby triggering a diversified warning method, accurately assessing the collision risk between the vehicle and the obstacle, and realizing accurate monitoring of obstacles in the blind spot, avoiding false warnings for stationary and harmless obstacles, minimizing interference to the driver, and improving driving experience and driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application Figure 1 ;

[0060] Figure 2A schematic diagram of an image processing method provided in an embodiment of the present application Figure 1 ;

[0061] Figure 3 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 2 ;

[0062] Figure 4 A schematic diagram of the fusion process of the first sub-fusion image and the second sub-fusion image provided in an embodiment of the present application;

[0063] Figure 5 A schematic diagram of the structure of an image processing device provided by the present application;

[0064] Figure 6 A schematic diagram of the structure of an image processing device provided in this application Figure 2 .

[0065] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0067] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0068] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0069] During driving, the vehicle's driving recorder will record the images of the vehicle while driving. In order to save storage space and improve usage efficiency, the data generated by the driving recorder is usually compressed. Existing solutions for compressing driving record data usually use H.264 video compression solutions and JPEG static image compression solutions.

[0070] However, the JPEG static image compression scheme is mainly used for compressing single-frame images. Although it provides a high compression ratio, the compressed image will have obvious image quality loss; although the H.264 video compression scheme can improve the image quality of compressed images, it needs to encode and decode the compressed images one by one, which increases the computational complexity of the codec and affects the coding efficiency.

[0071] Based on this, when the vehicle is in a scene where the landscape on both sides of the road does not change much, such as when the vehicle is driving on a highway or a corridor, the images recorded by the driving recorder are mostly similar images, and there are a lot of identical or similar contents in the similar images. At this time, if the existing image compression scheme is still used, a lot of identical or similar contents need to be repeatedly compressed, which will further affect the encoding efficiency.

[0072] Therefore, how to improve the image quality of compressed images while reducing the consumption of the encoding end and improving the encoding efficiency is the problem to be solved at present.

[0073] In response to the above problems, the present application provides an image processing method. When faced with multiple similar images, the method utilizes the correlation between similar images and adopts a distributed coding method to compress the multiple similar images, thereby reducing the amount of calculation on the encoding end and improving the coding efficiency. At the same time, the method can reconstruct the image based on the auxiliary information image and the decoded image, thereby reducing the blurriness of the reconstructed image and improving the image quality of the decoded image.

[0074] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0075] First, the structure of an image processing device provided in an embodiment of the present application is explained.

[0076] Figure 1 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application Figure 1 .like Figure 1 As shown, the image processing device 1 shown in this embodiment includes: an encoding end 11 and a decoding end 12.

[0077] The encoding end includes: an encoding module 111 , and the decoding end includes: a decoding module 121 .

[0078] The encoding module 111 is used to encode the original image;

[0079] The decoding module 121 is used to decode the original image after the encoding process and to decode the reference image.

[0080] It is understandable that the original image and the reference image are similar images. Since the original image and the reference image are similar images, there is a large amount of overlapping content in the original image and the reference image. When the image processing device provided in this embodiment performs compression processing, it only needs to encode the original image, and only decode the reference image.

[0081] After the original image and the reference image are decoded, the decoded first decoded image and the auxiliary information image may be fused to obtain a fused image, and the original image may be reconstructed based on the fused image.

[0082] The image processing device in this embodiment only needs to encode the original image, thereby reducing the amount of calculation on the encoding end and improving the encoding efficiency. At the same time, the device can reconstruct the image based on the auxiliary information image and the decoded image, thereby reducing the blurriness of the reconstructed image and improving the image quality of the decoded image.

[0083] In a possible implementation manner, the decoding end 12 further includes: a transformation module 122 .

[0084] The transformation module 122 is used to determine a target image block set corresponding to a plurality of image blocks in the first decoded image, and determine the target image block set as the auxiliary information image, wherein the target image block set is determined from the second decoded image.

[0085] It can be understood that using the first decoded image after decoding to transform the reference image can make the determined auxiliary information image more consistent with the original image, thereby improving the accuracy of the auxiliary information.

[0086] In a possible implementation, the transform module 122 is configured to perform block processing on the first decoded image and the second decoded image respectively to obtain a plurality of first image blocks corresponding to the first decoded image and a plurality of second image blocks corresponding to the second decoded image;

[0087] For any one of the plurality of first image blocks, determining a correlation between the first image block and the plurality of second image blocks;

[0088] Using the image block with the highest correlation among the multiple second image blocks as the target image block corresponding to the first image block;

[0089] The target image blocks corresponding to the first image blocks are used as the target image block set.

[0090] In a possible implementation manner, the decoding end 12 further includes: a multi-scale fusion module 123 .

[0091] The multi-scale fusion module 123 is used to perform expansion processing on the auxiliary information image and the first decoded image according to a plurality of preset resolutions to obtain a first sub-fusion image and a second sub-fusion image corresponding to each resolution;

[0092] For any one of the multiple resolutions, fusing the first sub-fusion image and the second sub-fusion image corresponding to the resolution to obtain a third sub-fusion image corresponding to the resolution;

[0093] An image pyramid fusion algorithm is used to perform fusion processing on a plurality of third sub-fusion images to obtain the fusion image.

[0094] In a possible implementation manner, the decoding end 12 further includes: an image reconstruction module 124 .

[0095] The image reconstruction module 124 is used to perform image reconstruction processing on the input fusion image and output a reconstruction result, wherein the reconstruction result includes: a reconstructed image corresponding to the original image;

[0096] Wherein, the image reconstruction module is trained based on historical original images and corresponding historical reconstructed images.

[0097] In a possible implementation, the image processing device further includes: an updating module 13;

[0098] The updating module 13 is used to determine whether the distortion weight reaches a preset weight;

[0099] When the distortion weight does not reach a preset weight, the encoding parameters of the encoding end are updated based on the distortion weight.

[0100] Figure 2 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 1 The execution subject of this embodiment can be, for example, Figure 1 The image processing device 1 shown in the embodiment. Figure 2 As shown, the image processing method provided in this embodiment includes:

[0101] S201. Perform encoding and decoding processing on an original image through an encoding end and a decoding end to obtain a first decoded image.

[0102] The original image and the reference image described below are both images captured by a driving recorder on the vehicle. The two are similar images.

[0103] In this step, the original image may be, for example, an image containing sufficient image details. The purpose of using the encoder at the encoding end and the decoder at the decoding end to encode and decode the original image is to provide sufficient original information for the subsequent auxiliary information image, so as to make the auxiliary information more accurate.

[0104] It is understandable that when encoding and decoding the original image, it is necessary to fully consider the relationship between the compression rate and the distortion rate of the original image, so as to ensure that the compressed image occupies a smaller memory and that the compressed image is not distorted.

[0105] In a possible implementation, after encoding and decoding the original image, the quantized latent vector corresponding to the original image is obtained. The decoder can quantize the latent vector Get the reconstructed image corresponding to the original image.

[0106] In this step, the relationship between the compression rate and the distortion rate may be determined by minimizing the distortion function. The relationship between the compression rate and the distortion rate may be expressed, for example, by the following formula:

[0107]

[0108] in, Represents the quantized latent vector The entropy of the latent vector bit cost), represents the distortion function and β is a scalar that sets the trade-off between the two.

[0109] S202: The decoding end performs transformation processing on the reference image to obtain an auxiliary information image.

[0110] The number of reference images may be one or more, as long as they are similar images to the original image. In the case where there are multiple reference images, the processing of the multiple reference images is similar, and the following example is taken as an example of one reference image.

[0111] The auxiliary information image may represent, for example, an image that provides rich detail information for the original image.

[0112] In this step, the purpose of transforming the reference image is to subsequently improve the details in the original image.

[0113] It is understandable that in this embodiment, the encoding end only needs to encode the original image, and does not need to encode the reference image. When multiple similar images are processed, since only part of the images need to be encoded, the amount of calculation at the encoding end is reduced, and the encoding efficiency is improved.

[0114] S203: Fusing the auxiliary information image and the first decoded image to obtain a fused image.

[0115] The fused image includes many image details of the first decoded image and auxiliary information in the auxiliary information image, and has higher definition and lower distortion rate than the first decoded image.

[0116] In this step, for example, a portion of the auxiliary information image with higher definition may be merged into the first decoded image to improve the definition of the first decoded image.

[0117] S204: Determine a reconstructed image corresponding to the original image based on the fused image.

[0118] Since the fused image includes many image details in the first decoded image, image reconstruction processing can be performed based on the fused image to obtain a reconstructed image corresponding to the original image.

[0119] It can be understood that the reconstructed image is compressed relative to the original image, and because the reconstructed image is reconstructed based on the auxiliary information image, it has a lower distortion rate and a higher definition.

[0120] The present embodiment provides an image processing method. When facing multiple similar images, the method performs encoding and decoding processing on the original image through the encoding end and the decoding end to obtain a first decoded image; the decoding end performs transformation processing on the reference image to obtain an auxiliary information image; the auxiliary information image and the first decoded image are fused to obtain a fused image; based on the fused image, a reconstructed image corresponding to the original image is determined; since the method only needs to encode the original image, the calculation amount of the encoding end is reduced and the encoding efficiency is improved. At the same time, the method can reconstruct the image based on the auxiliary information image and the decoded image, thereby reducing the blur of the reconstructed image and improving the image quality of the decoded image.

[0121] Figure 3 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 2 In this embodiment, the decoding end of the image processing device includes: a decoding module, a transformation module, a multi-scale fusion module and an image reconstruction module. Figure 2Based on the embodiment, a possible implementation of the image processing method is described in detail. The image processing method provided in this embodiment includes:

[0122] S301. Perform encoding and decoding processing on an original image through the encoding end and the decoding end to obtain a first decoded image.

[0123] Among them, step S301 is similar to the above step S201 and will not be repeated here.

[0124] S302: Decode the reference image through the decoding module to obtain a second decoded image.

[0125] The purpose of decoding the reference image is to make the reference image more closely matched with the first decoded image after decoding.

[0126] It can be understood that the reference image is not encoded but only decoded.

[0127] The second decoded image after decoding has the same resolution as the first decoded image, so that the second decoded image and the first decoded image have a higher matching degree when they are subsequently transformed.

[0128] 303. Determine, by means of a transformation module, a target image block set corresponding to a plurality of image blocks in the first decoded image, and determine the target image block set as an auxiliary information image.

[0129] The target image block set is determined from the second decoded image.

[0130] In this step, the first decoded image and the second decoded image may be firstly block processed according to the corresponding block strategy, and then the corresponding target image blocks in the second decoded image may be determined based on the image blocks after the first decoded image is block-processed, thereby obtaining an auxiliary information image.

[0131] In a possible implementation manner, a specific implementation manner of determining the target image block set corresponding to the multiple image blocks in the first decoded image in step S303 may include, for example:

[0132] S3031. Perform block processing on the first decoded image and the second decoded image respectively to obtain a plurality of first image blocks corresponding to the first decoded image and a plurality of second image blocks corresponding to the second decoded image.

[0133] Since the vehicle is in a driving state, the image contents contained in the original image captured by the driving recorder are different from those contained in the reference image.

[0134] After the first decoded image is processed by block division, a plurality of first image blocks are obtained; after the second decoded image is processed by block division, a plurality of second image blocks are obtained.

[0135] It is understandable that there may be overlapping parts in the first image blocks and the second image blocks. For example, the recorded picture in the first decoded image may also appear in the second decoded image. That is, the content captured by the original image and the reference image may be the same.

[0136] S3032. For any one first image block among the plurality of first image blocks, determine a correlation between the first image block and the plurality of second image blocks.

[0137] For any one of the multiple first image blocks, it is necessary to first determine a second image block similar to the first image block, so the correlation between the first image block and the multiple image blocks may be determined first.

[0138] S3033: Use the image block with the highest correlation among the multiple second image blocks as the target image block corresponding to the first image block.

[0139] The target image block may, for example, represent a second image block that is most similar to the corresponding first image block.

[0140] After the correlation between the first image block and the plurality of second image blocks is obtained, the second image block with the highest correlation may be used as the target image block corresponding to the first image block.

[0141] It can be understood that when determining the target image block, the second decoded image can be mapped to the same plane as the first decoded image. Each non-overlapping first image block in the first decoded image is compared with all possible related second image blocks in the second decoded image, and then the second image block with the highest correlation among all possible related second image blocks is selected as the target image block of the corresponding first image block.

[0142] In a possible implementation, when determining the target image block, the following formula may be used:

[0143]

[0144] Among them, corr() represents the correlation between image blocks, π(X dec (i)) represents the image blocks around the first image block, π(Y dec (j)) represents the image blocks surrounding the second image block.

[0145] In the above formula, the correlation between the first image block and the second image block can be determined based on the similarity between the image blocks around the first image block and the image blocks around the second image block, and then the second image block with the highest correlation is used as the target image block of the first image block.

[0146] S3034. Use the multiple target image blocks corresponding to the multiple first image blocks as the target image block set.

[0147] After obtaining the target image block corresponding to each first image block, the multiple target image blocks can be used as an image block set.

[0148] It can be understood that the target image block set may include, for example, multiple target image blocks, and may also include position information of first image blocks corresponding to the multiple target image blocks in the first decoded image. This application does not impose any limitation on this.

[0149] In a possible implementation, the target image block set may be expressed by, for example, the following formula:

[0150] Y syn (i) = Y(f(i))

[0151] Among them, f() is a sparse function.

[0152] 304. Use a multi-scale fusion module to perform expansion processing on the auxiliary information image and the first decoded image according to multiple preset resolutions to obtain a first sub-fusion image and a second sub-fusion image corresponding to each resolution.

[0153] The purpose of performing the expansion processing according to the preset resolution is to increase the fusion samples so as to make the details and texture information of the subsequent fused image more accurate, thereby making the fused image more accurate.

[0154] It can be understood that in this step, for example, the image can be expanded according to a resolution of 1 / 2 or 1 / 4 of the original size. The expanded first sub-fused image and the second sub-fused image are the first decoded image corresponding to the original resolution, 1 / 2 resolution, and 1 / 4 resolution, and the second decoded image corresponding to the original resolution, 1 / 2 resolution, and 1 / 4 resolution.

[0155] S305 . For any one of the multiple resolutions, perform fusion processing on the first sub-fusion image and the second sub-fusion image corresponding to the resolution to obtain a third sub-fusion image corresponding to the resolution.

[0156] When the sub-fusion images are fused, it is necessary to fuse the first sub-fusion image and the second sub-fusion image with the same resolution.

[0157] Figure 4 A schematic diagram of the fusion process of the first sub-fused image and the second sub-fused image provided in an embodiment of the present application.

[0158] like Figure 4 As shown, the first decoded image X dec After expansion, we get X′ and X1′ / 2 and X1′ / 4 The first sub-fused images of three different resolutions; the second decoded image Y syn After expansion, we get Y′ and Y1′ / 2 and Y1′ / 4 Second sub-fused images at three different resolutions.

[0159] When performing the fusion process, X′ and Y′ are fused, X1′ / 2 and Y1′ / 2 Fusion, X1′ / 4 and Y1′ / 4 After fusion processing, we can get I f1 ,I f1 / 2 and I f1 / 4 The third sub-fused image corresponding to the three resolutions.

[0160] It can be understood that fusing sub-fused images with the same resolution can make the resolution of the obtained third sub-fused image clearer.

[0161] S306 , using an image pyramid fusion algorithm to perform fusion processing on the plurality of third sub-fusion images to obtain the fusion image.

[0162] Among them, continue to refer to Figure 4 The content shown in I f1 ,I f1 / 2 and I f1 / 4 After the third sub-fusion images corresponding to the three resolutions are obtained, the image pyramid fusion algorithm can be used to fuse the three third sub-fusion images again to obtain a fused image I f .

[0163] S307: Input the fused image into an image reconstruction module to obtain a reconstruction result output by the image reconstruction module.

[0164] The reconstruction result includes: a reconstructed image corresponding to the original image. The image reconstruction module is trained based on the historical original image and the corresponding historical reconstructed image.

[0165] The image reconstruction module is used to perform reconstruction processing based on the input fused image to obtain a reconstructed image corresponding to the original image.

[0166] The image reconstruction module can be trained based on historical original images and corresponding historical reconstructed images. The image reconstruction module can include a full convolutional network, which is used to analyze and process the fused image to obtain corresponding associated information, and then output a reconstructed image based on the associated information. The specific implementation process can be implemented by referring to the existing convolutional reconstruction network.

[0167] In a possible implementation manner, the reconstruction result further includes: a distortion weight.

[0168] The distortion weight is used to indicate the degree of distortion of the reconstructed image relative to the original image.

[0169] After the image reconstruction module outputs the reconstructed image and the distortion weight corresponding to the reconstructed image, it can be determined whether the distortion weight reaches a preset weight.

[0170] If the distortion weight does not reach the preset weight, it indicates that the output reconstructed image is not seriously distorted and meets the definition requirement of the original image. At this time, the encoding parameters of the encoding end can be updated based on the distortion weight. The encoding parameters may include compression parameters, and different compression parameters correspond to different compression rates.

[0171] The update process may include, for example, determining a current encoding parameter of the encoding end and increasing a usage weight of the encoding parameter.

[0172] If the distortion weight reaches the preset weight, it indicates that the output reconstructed image is seriously distorted and cannot meet the definition requirement of the original image. At this time, the usage weight of the encoding parameter of the encoding end can be reduced based on the distortion weight.

[0173] The present embodiment provides an image processing method, which determines a target image block set from a second decoded image based on a first decoded image, and uses the target image block set as an auxiliary information image, thereby making the auxiliary information more closely fit the original image and improving the accuracy of the auxiliary information; and since only the original image needs to be encoded, the amount of calculation at the encoding end is reduced and the encoding efficiency is improved; the method also fuses the auxiliary information image with the first decoded image, that is, the auxiliary information image is used to complete and correct the first decoded image, thereby reducing the blurriness of the reconstructed image and improving the image quality of the decoded image.

[0174] Figure 5 The structure diagram of an image processing device provided in an embodiment of the present application is shown in FIG. The image processing device provided in this embodiment is applied to an image processing device, and the image processing device includes: an encoding end and a decoding end. Figure 5 As shown, the image processing device 500 provided in this embodiment includes:

[0175] The processing module is used to perform encoding and decoding processing on the original image through the encoding end and the decoding end to obtain a first decoded image; perform transformation processing on the reference image through the decoding end to obtain an auxiliary information image; perform fusion processing on the auxiliary information image and the first decoded image to obtain a fused image; and determine a reconstructed image corresponding to the original image based on the fused image.

[0176] An image processing device provided in this embodiment can execute the image processing method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail in this embodiment.

[0177] Figure 6 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application Figure 2 .like Figure 6 As shown, the image processing device 600 may include at least one processor 601, at least one storage medium 602, and a communication interface 603. Among them, the at least one processor 601 is used to implement the image processing method provided by the above embodiment of the present application.

[0178] At least one memory 602 is used to store program instructions and / or data. The memory 602 is coupled to the processor 601. The coupling in the embodiment of the present application is an indirect coupling or communication connection between devices, units or modules, which can be electrical, mechanical or other forms, and is used for information exchange between devices, units or modules. The processor 601 may operate in conjunction with the memory 602. The processor 601 may execute program instructions stored in the memory 602. At least one of the at least one memory may be included in the processor.

[0179] The communication interface 603 is used to communicate with other devices through a transmission medium, so that the image processing device 600 can communicate with other devices. The communication interface 603 can be, for example, a transceiver, an interface, a bus, a circuit, or a device capable of implementing a transceiver function. The processor 601 can use the communication interface 603 to send and receive data and / or information, and is used to implement the method provided in the above embodiment of the present application. Please refer to the detailed description in the previous embodiment for details, which will not be repeated here.

[0180] The specific connection medium between the processor 601, the memory 602 and the communication interface 603 is not limited in the embodiment of the present application. Figure 6 In the embodiment, the processor 601, the memory 602 and the communication interface 603 are connected via a bus 604. The bus 604 is Figure 6 The connection between other components is only for illustration and is not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0181] It should be understood that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined to perform. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0182] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0183] A computer-readable storage medium is also provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the technical solution of the above-mentioned image processing method embodiment is implemented. The implementation principle and technical effect are similar and will not be repeated here.

[0184] A computer program product is also provided in an embodiment of the present application, including a computer program. When the computer program is executed by a processor, the technical solution of the above-mentioned image processing method embodiment is implemented. The implementation principle and technical effect are similar and will not be repeated here.

[0185] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in combination with computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The steps in the several embodiments provided in this application can be implemented by different execution units or execution modules, or by different devices. It should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the method embodiments described above are merely illustrative. For example, multiple steps in the method embodiments can be implemented by multiple units or modules of the corresponding apparatus. The division of units in the apparatus is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, which may be electrical, mechanical or other forms.

[0186] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0188] In the above-described embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the flow or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0189] If this function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0190] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0191] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to an image processing device, the image processing device includes: an encoding end and a decoding end, and the method includes: The encoding and decoding end perform encoding and decoding processing on the original image to obtain a first decoded image; The reference image is transformed by the decoding end to obtain an auxiliary information image; fusing the auxiliary information image and the first decoded image to obtain a fused image; Based on the fused image, a reconstructed image corresponding to the original image is determined.

2. The method according to claim 1, characterized in that: The decoding end includes: a decoding module and a transformation module, and the decoding end transforms the reference image to obtain the auxiliary information image, including: Decoding the reference image through the decoding module to obtain a second decoded image; A target image block set corresponding to a plurality of image blocks in the first decoded image is determined through the transformation module, and the target image block set is determined as the auxiliary information image, wherein the target image block set is determined from the second decoded image.

3. The method according to claim 2, characterized in that The determining a target image block set corresponding to a plurality of image blocks in the first decoded image comprises: respectively performing block processing on the first decoded image and the second decoded image to obtain a plurality of first image blocks corresponding to the first decoded image and a plurality of second image blocks corresponding to the second decoded image; For any one of the plurality of first image blocks, determining a correlation between the first image block and the plurality of second image blocks; Using the image block with the highest correlation among the plurality of second image blocks as the target image block corresponding to the first image block; A plurality of target image blocks corresponding to the plurality of first image blocks are taken as the target image block set.

4. The method according to claim 3, characterized in that The decoding end further includes: a multi-scale fusion module, wherein the fusion processing of the auxiliary information image and the first decoded image to obtain a fused image includes: Using the multi-scale fusion module, the auxiliary information image and the first decoded image are expanded according to a plurality of preset resolutions to obtain a first sub-fusion image and a second sub-fusion image corresponding to each resolution; For any one of the multiple resolutions, fusing the first sub-fusion image and the second sub-fusion image corresponding to the resolution to obtain a third sub-fusion image corresponding to the resolution; An image pyramid fusion algorithm is used to perform fusion processing on a plurality of third sub-fusion images to obtain the fusion image.

5. The method according to claim 4, characterized in that The decoding end further includes: an image reconstruction module, which determines the reconstructed image corresponding to the original image based on the fused image, including: Inputting the fused image into the image reconstruction module to obtain a reconstruction result output by the image reconstruction module, wherein the reconstruction result includes: a reconstructed image corresponding to the original image; Wherein, the image reconstruction module is trained based on historical original images and corresponding historical reconstructed images.

6. The method according to claim 5, characterized in that The reconstruction result also includes: a distortion weight, and the method further includes: Determining whether the distortion weight reaches a preset weight; When the distortion weight does not reach a preset weight, the encoding parameters of the encoding end are updated based on the distortion weight.

7. An image processing device, characterized in that: Applied to an image processing device, the image processing device comprises: an encoding end and a decoding end, the device comprises: The processing module is used to perform encoding and decoding processing on the original image through the encoding end and the decoding end to obtain a first decoded image; perform transformation processing on the reference image through the decoding end to obtain an auxiliary information image; perform fusion processing on the auxiliary information image and the first decoded image to obtain a fused image; and determine a reconstructed image corresponding to the original image based on the fused image.

8. An image processing device, characterized in that: include: Encoding end and decoding end; The encoding end is used to encode the original image; The decoding end is used to decode the original image after the encoding process to obtain a first decoded image; The decoding end is further used to transform the reference image to obtain an auxiliary information image; fuse the auxiliary information image and the first decoded image to obtain a fused image; and determine a reconstructed image corresponding to the original image based on the fused image.

9. An image processing device, characterized in that: include: Memory, processor and communication interface; Wherein, the memory stores computer-executable instructions, and the communication interface is used to communicate with other devices via a transmission medium; The processor executes the computer-executable instructions stored in the memory to implement the image processing method according to any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The computer storage medium stores computer execution instructions, which are used to implement the image processing method according to any one of claims 1 to 6 when executed by a processor.

Images