Image encoding and decoding method and related products

Abstract

The embodiment of the present application discloses an image encoding and decoding method and related products. The method can generate a target imaging model based on a target area containing a light source in an image to be encoded, obtain a simulated light source image based on the target imaging model, and then encode and decode a first image obtained after removing the simulated light source image from the image to be encoded and the model parameters of the target imaging model, obtain the target imaging model based on the model parameters, and then input the pixel coordinates of the target area into the target imaging model respectively to obtain the simulated light source image, and finally compensate the first image based on the simulated light source image to obtain the original image. The method can solve the problem of a large amount of storage resources and computing resources of the encoder and decoder being occupied during the encoding and decoding process of an image carrying a light source by removing the simulated light source image from the image to be encoded and then encoding and decoding the simulated light source image and the removed image respectively.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to an image encoding and decoding method and related products. Background Art

[0002] In a video sequence carrying a light source, since some image blocks in the image carry light information and the light information is different in each frame of the video sequence, the video sequence will occupy a large amount of memory space.

[0003] At present, when encoding an image carrying a light source, the obtained data stream contains a large amount of information, which will occupy a large amount of storage resources and computing resources of the encoder and decoder during the encoding and decoding processes of the encoder and decoder, thereby affecting the working performance of the encoder and decoder. Summary of the invention

[0004] The embodiments of the present application provide an image encoding and decoding method and related products, which can optimize the encoding and decoding of images carrying light sources, thereby solving the problem of a large amount of storage resources and computing resources of the encoder and decoder being occupied during the encoding process of the encoder and the decoding process of the decoder.

[0005] In a first aspect, an embodiment of the present application provides an image encoding method, the method comprising:

[0006] Obtaining an image to be encoded;

[0007] Determining a target area containing a light source in the image to be encoded;

[0008] Using all pixel points in the target area of ​​the image to be encoded as samples to train an initial imaging model to obtain a target imaging model, wherein the target imaging model is used to generate pixel values ​​according to input pixel point coordinates;

[0009] Inputting the pixel coordinates of the target area into the target imaging model respectively to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the image to be encoded;

[0010] Eliminating the simulated light source image from the image to be encoded to obtain a first image;

[0011] The first image, the indication information for indicating the target area, and the model parameters of the target imaging model are encoded respectively to obtain a second image, the encoded indication information, and the encoded model parameters.

[0012] In a possible implementation manner, determining a target area containing a light source in the image to be encoded includes:

[0013] Determining the light source from the image to be encoded;

[0014] A rectangular region of a target area centered on the light source is determined as the target region.

[0015] In a possible implementation manner, the indication information is a coordinate value of the light source.

[0016] In a possible implementation, the training of the initial imaging model using all pixels in the target area in the image to be encoded as samples to obtain the target imaging model includes:

[0017] Inputting each pixel point in the target area of ​​the image to be encoded into the initial imaging model to obtain a predicted value of each pixel point;

[0018] Taking the difference between the pixel value of each pixel in the to-be-encoded image and the predicted value of each pixel as the error of each pixel;

[0019] Sum the errors of all pixels in the target area to obtain a total difference;

[0020] The initial imaging model is updated according to the total difference to obtain a target imaging model.

[0021] In a possible implementation manner, removing the simulated light source image from the image to be encoded to obtain the first image includes:

[0022] The pixel value of the first pixel point in the first image is determined as the difference between the pixel value of the first pixel point in the image to be encoded and the pixel value of the pixel point corresponding to the first pixel point in the simulated light source image, and the first pixel point is a pixel point in the target area.

[0023] In a possible implementation, the method further includes:

[0024] Determine a difference between a pixel value of a second pixel point in the image to be encoded and a pixel value of a pixel point corresponding to the second pixel point in the simulated light source image as a first difference value, wherein the second pixel point is any pixel point in the target area;

[0025] When the first difference is smaller than a target value, the second pixel is determined as the first pixel.

[0026] In a second aspect, an embodiment of the present application provides an image decoding method, the method comprising:

[0027] Decoding the second image, the encoded indication information, and the encoded model parameters respectively to obtain the first image, the indication information for indicating the target area, and the model parameters;

[0028] Update the initial imaging model according to the model parameters to obtain a target imaging model;

[0029] According to the indication information, obtaining pixel coordinates of the target area;

[0030] Inputting the pixel coordinates of the target area into the target imaging model respectively to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the image to be encoded;

[0031] The first image is compensated according to the simulated light source image to obtain an original image.

[0032] In a possible implementation manner, compensating the first image according to the simulated light source image to obtain an original image includes:

[0033] The pixel value of the first pixel point in the original image is determined as the sum of the pixel value of the first pixel point in the first image and the pixel value of the pixel point corresponding to the first pixel point in the simulated light source image, and the first pixel point is a pixel point in the target area.

[0034] In a possible implementation, the method further includes:

[0035] When a pixel value of a second pixel point in the first image is less than a target value, the second pixel point is determined as the first pixel point, and the second pixel point is any pixel point in the target area.

[0036] In a third aspect, an embodiment of the present application provides an encoder, the encoder comprising a unit for executing the method of the first aspect. Specifically, the device may include:

[0037] A first acquisition unit, used to acquire an image to be encoded;

[0038] a determination unit, configured to determine a target area containing a light source in the image to be encoded;

[0039] A training unit, used to train an initial imaging model using all pixel points in the target area in the image to be encoded as samples to obtain a target imaging model, wherein the target imaging model is used to generate pixel values ​​according to input pixel point coordinates;

[0040] A simulation unit, used for inputting the pixel coordinates of the target area into the target imaging model respectively to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the image to be encoded;

[0041] A generating unit, configured to remove the simulated light source image from the image to be encoded to obtain a first image;

[0042] The encoding unit is used to encode the first image, the indication information indicating the target area and the model parameters of the target imaging model respectively to obtain the second image, the encoded indication information and the encoded model parameters.

[0043] In a fourth aspect, an embodiment of the present application provides a decoder, the decoder comprising a unit for executing the method of the second aspect. Specifically, the device may include:

[0044] A decoding unit, used to decode the second image, the encoded indication information and the encoded model parameters respectively to obtain the first image, the indication information for indicating the target area and the model parameters;

[0045] An updating unit, used for updating the initial imaging model according to the model parameters to obtain a target imaging model;

[0046] An acquisition unit, used for acquiring pixel coordinates of the target area according to the indication information;

[0047] A simulation unit is used to input the pixel coordinates of the target area into the target imaging model to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the image to be encoded.

[0048] A compensation unit is used to compensate the first image according to the simulated light source image to obtain an original image.

[0049] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program that supports an encoder or a decoder to execute the above-mentioned method, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the above-mentioned methods of the first and second aspects.

[0050] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the methods of the first and second aspects described above.

[0051] In a seventh aspect, an embodiment of the present application provides a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the methods of the first and second aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0053] Figure 1 It is a structural schematic diagram of an image encoding and decoding system provided in an embodiment of the present application;

[0054] Figure 2 It is a flowchart of an image encoding method provided by an embodiment of the present application;

[0055] Figure 3 It is a flowchart of an image decoding method provided by an embodiment of the present application;

[0056] Figure 4 is a schematic diagram of the structure of an encoder 30 provided in an embodiment of the present application;

[0057] Figure 5 is a schematic diagram of the structure of a decoder 40 provided in an embodiment of the present application;

[0058] Figure 6 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0059] Figure 7 It is a structural diagram of a decoder provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0061] It should be understood that the terms "first", "second", and "third" in the claims, specifications, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0062] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0063] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0064] In order to better understand the image encoding and decoding method and related products provided in the embodiments of the present application, the embodiments of the present application provide an image encoding and decoding system. Figure 1 , Figure 1 1 is a schematic diagram of the structure of an image encoding and decoding system provided in an embodiment of the present application, and the system may include an encoder 100 and a decoder 200. Among them:

[0065] Among them, the encoder 100 can obtain an image to be encoded; determine a target area containing a light source in the above-mentioned image to be encoded; use all pixel points in the above-mentioned target area in the above-mentioned image to be encoded as samples to train an initial imaging model to obtain a target imaging model, and the above-mentioned target imaging model is used to generate pixel values ​​according to the input pixel point coordinates; input the pixel point coordinates of the above-mentioned target area into the above-mentioned target imaging model respectively to obtain a simulated light source image, and the pixel points in the above-mentioned simulated light source image correspond one-to-one to the pixel points in the above-mentioned target area in the above-mentioned image to be encoded; remove the above-mentioned simulated light source image from the above-mentioned image to be encoded to obtain a first image; respectively encode the above-mentioned first image, the indication information used to indicate the above-mentioned target area and the model parameters of the above-mentioned target imaging model to obtain a second image, the encoded indication information and the encoded model parameters.

[0066] After processing the image to be encoded, the encoder 100 may send the obtained second image, the encoded indication information and the encoded model parameters to the decoder 200 .

[0067] Among them, the decoder 200 can decode the second image, the encoded indication information and the encoded model parameters respectively to obtain the first image, the indication information for indicating the target area and the model parameters; update the initial imaging model according to the above model parameters to obtain the target imaging model; obtain the pixel coordinates of the above target area according to the above indication information; input the pixel coordinates of the above target area into the above target imaging model respectively to obtain a simulated light source image, and the pixels in the above simulated light source image correspond one-to-one to the pixels in the above target area in the above image to be encoded; compensate the above first image according to the above simulated light source image to obtain the original image.

[0068] It should be noted that Figure 1The system architecture of the image coding and decoding system in the embodiment of the present application is only an exemplary implementation manner. The system architecture of the image coding and decoding system in the embodiment of the present application includes but is not limited to the above system architecture.

[0069] Combine the following Figure 1 The flowchart of an image encoding method provided by an embodiment of the present application is shown, which specifically describes the image encoding method of the embodiment of the present application. The method may include but is not limited to the following steps:

[0070] S101: Obtain an image to be encoded.

[0071] Specifically, the encoder may obtain a frame of image from the video as the image to be encoded. For example, the server may use all images in the video as the images to be encoded, and execute the image encoding method for each frame of the video.

[0072] S102: Determine a target area including a light source in the image to be encoded.

[0073] Specifically, the encoder may determine the light source from the image to be encoded according to the light source detection algorithm, and determine a rectangular region of the target area centered on the light source as the target region.

[0074] In one implementation, for each pixel in the image to be encoded, the encoder can determine the area corresponding to the pixel from the area of ​​the target area centered on the pixel, and then determine whether the pixel values ​​of all pixels in the area corresponding to the pixel meet the target threshold. For example, when the pixel value of the area corresponding to pixel number one meets the target threshold, it is determined that pixel number one is a light source, and the area corresponding to pixel number one is the above-mentioned target area containing the light source.

[0075] It should be noted that, in some embodiments, when the encoder detects that the image to be encoded has no light source, the following steps are not executed, and other encoding processing can be performed on the image to be encoded, which is not described in detail in the embodiments of the present application.

[0076] S103, training an initial imaging model using all pixel points in a target area of ​​the image to be encoded as samples to obtain a target imaging model, wherein the target imaging model is used to generate pixel values ​​according to input pixel point coordinates.

[0077] Specifically, the encoder can obtain the coordinate values ​​and pixel values ​​of all pixels in the target area, and then input each pixel in the target area in the image to be encoded into the initial imaging model to obtain the predicted value of each pixel, and then use the difference between the pixel value of each pixel in the image to be encoded and the predicted value of each pixel as the error of each pixel, sum the errors of all pixels in the target area to obtain the total difference, optimize the initial imaging model according to the total difference, and obtain the target imaging model when the total difference is minimum.

[0078] It should be noted that the difference between the pixel value of the pixel in the above-mentioned image to be encoded and the predicted value of the pixel is the absolute value of the difference between the pixel value of the pixel in the above-mentioned image to be encoded and the predicted value of the pixel.

[0079] The above initial imaging model may be a variation of a two-dimensional Gaussian function, and the formula is as follows:

[0080]

[0081] Among them, x, y are the horizontal and vertical coordinates of the pixel point, and g(x, y) is the predicted value of the pixel point. The model parameters of the imaging model include α, β and σ. When the above total difference is the smallest, the optimal model parameters can be obtained.

[0082] The imaging model may also be other functions, which are not limited here. In some embodiments, the parameters of the imaging model may also include a target value for screening pixels to be removed in the target area and the area size of the target area, which are not limited here.

[0083] It should be noted that when the image to be encoded includes multiple light sources, the encoder can determine the target area corresponding to each light source according to each light source, and determine the target imaging model corresponding to the area according to the target area corresponding to each light source. The specific process of obtaining the target imaging model of each light source can refer to the above-mentioned related content. Among them, the imaging model used by each light source can be different, which is not limited here.

[0084] S104, inputting the pixel coordinates of the target area into the target imaging model respectively to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area of ​​the image to be encoded.

[0085] Specifically, the encoder inputs the coordinates of each pixel in the target area into the target imaging model, and obtains the prediction value corresponding to each pixel. According to the prediction values ​​of all pixels, a simulated light source image can be generated. It can be understood that the simulated light source image generated by the encoder according to the target area is equal to the size of the target area, and the simulated light source image corresponds one-to-one to the pixels of the target area in the image to be encoded.

[0086] In some embodiments, the encoder may generate a simulated light source image of the same size as the image to be encoded. After obtaining the predicted value of each pixel in the image to be encoded according to the target imaging model, the encoder may input the predicted value of the third pixel at the position of the coordinate value of the third pixel in a blank image of the same size as the image to be encoded, wherein the pixel value of each pixel in the blank image is zero, and the third pixel is any pixel in the target area of ​​the image to be encoded, and finally, a simulated light source image of the same size as the image to be encoded is obtained.

[0087] It should be noted that when the image to be encoded contains multiple light sources, the encoder can generate a simulated light source image corresponding to each light source according to the target area corresponding to each light source and the imaging model corresponding to each light source.

[0088] S105: Eliminate the simulated light source image from the image to be encoded to obtain a first image.

[0089] Specifically, the encoder can determine the pixel value of the first pixel in the above-mentioned first image as the difference between the pixel value of the above-mentioned first pixel in the above-mentioned image to be encoded and the pixel value of the pixel corresponding to the above-mentioned first pixel in the above-mentioned simulated light source image, and the above-mentioned first pixel is a pixel in the above-mentioned target area.

[0090] In one implementation, the encoder can also determine the difference between the pixel value of the second pixel point in the above-mentioned image to be encoded and the pixel value of the pixel point corresponding to the above-mentioned second pixel point in the above-mentioned simulated light source image as the first difference, and the above-mentioned second pixel point is any pixel point in the above-mentioned target area. When the above-mentioned first difference is less than the target value, the above-mentioned second pixel point is determined as the above-mentioned first pixel point.

[0091] The formula for calculating the pixel points of the first image may be as follows:

[0092]

[0093] Among them, B ij is the pixel value of the second pixel in the image to be encoded, A ij is the pixel value of the pixel corresponding to the second pixel in the simulated light source image, B′ ijis the pixel value of the second pixel in the first image, and ε is the target value mentioned above, which can be generated by the initial imaging model and the pixel points of the target area mentioned above.

[0094] It should be noted that the first image includes multiple light sources, and the server can remove the simulated light source images generated by the multiple light sources from the above-mentioned image to be encoded according to the above-mentioned related content to obtain the first image.

[0095] S106 , respectively encode the first image, the indication information for indicating the target area, and the model parameters of the target imaging model to obtain the second image, the encoded indication information, and the encoded model parameters.

[0096] The encoder may encode different data in different ways. For example, the method for encoding the model parameters may be encoding with reference to a codebook. Since the encoder and decoder store some commonly used imaging model parameters (i.e., codebooks), assuming that the estimated value of the imaging model parameter is C, and Ci is the i-th imaging model stored by the encoder, the encoder may encode i and C-Ci.

[0097] In some embodiments, when there are multiple target areas, the encoder may encode the number of the target areas, so that the decoder obtains the multiple target areas according to the indication information when decoding to obtain the number of the target areas.

[0098] It can be understood that a video is an image sequence composed of multiple frames of images. When a decoder encodes a video, it can perform the above steps on each frame of the video to achieve encoding of the entire video.

[0099] Combine the following Figure 7 The flowchart of an image decoding method provided by an embodiment of the present application is shown, which specifically describes the image decoding method of the embodiment of the present application. The method may include but is not limited to the following steps:

[0100] S201. Decode the second image, the encoded indication information, and the encoded model parameters respectively to obtain the first image, the indication information for indicating the target area, and the model parameters.

[0101] Specifically, the decoding method of the decoder corresponds one-to-one to the encoding method of the encoder.

[0102] S202: Update the initial imaging model according to the model parameters to obtain a target imaging model.

[0103] It should be understood that the initial imaging model used by the decoder is the same as the initial imaging model used by the encoder.

[0104] S203: Obtain pixel coordinates of the target area according to the indication information.

[0105] Specifically, the decoder can determine the target pixel point according to the rules agreed with the decoder. For example, the rule for determining the target area is that the area with a radius R centered on the light source is the target area. After the decoder decodes the above indication information, it can obtain the coordinate value of the light source. Then, the decoder can determine the target area based on the coordinate value of the light source.

[0106] S204, inputting the pixel coordinates of the target area into the target imaging model respectively to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area of ​​the image to be encoded.

[0107] Specifically, the decoder inputs the coordinates of each pixel point in the above-mentioned target area into the above-mentioned target imaging model respectively, and can obtain the prediction value corresponding to each pixel point. According to the prediction values ​​of all pixel points, a simulated light source image can be generated, and the simulated light source image corresponds one-to-one to the pixel points of the target area in the above-mentioned image to be encoded.

[0108] In an embodiment where multiple target areas exist, the decoder may determine an imaging model of each target area according to the target area, and then obtain a simulated light source image corresponding to the target area according to the target area and the imaging model corresponding to the target area.

[0109] S205 . Compensate the first image according to the simulated light source image to obtain an original image.

[0110] Specifically, the decoder can determine the pixel value of the first pixel in the original image as the sum of the pixel value of the first pixel in the first image and the pixel value of the pixel corresponding to the first pixel in the simulated light source image, where the first pixel is a pixel in the target area.

[0111] In one implementation, the decoder may also determine the second pixel point as the first pixel point when the pixel value of the second pixel point in the first image is less than the target value, and the second pixel point is any pixel point in the target area. Specifically, the decoder may compare the pixel value of each pixel point in the target area of ​​the first image with the target value, and determine the pixel point as the first pixel point when the pixel value of the pixel point is less than the target value.

[0112] The formula for calculating the pixel value of the original image can be as follows:

[0113]

[0114] Among them, B ij is the pixel value of the second pixel in the first image, A ijis the pixel value of the pixel corresponding to the second pixel in the simulated light source image, C ij is the pixel value of the second pixel in the original image, and ε is the target value mentioned above, which can be generated by the initial imaging model and the pixel points of the target area mentioned above.

[0115] In some embodiments, when the first image includes multiple light sources, the decoder can process each light source in turn according to a specified order. Specifically, each target area containing the light source is compensated according to a simulated light source image corresponding to the light source, and finally, the original image is obtained.

[0116] It can be understood that a video is an image sequence composed of multiple frames of images. When a decoder decodes a video, it can perform the above steps on each frame of the video to achieve decoding of the entire video.

[0117] like Figure 4 As shown, Figure 4 : is a schematic diagram of the structure of an encoder 30 provided in an embodiment of the present application. The encoder 30 may at least include:

[0118] A first acquisition unit 301 is used to acquire an image to be encoded;

[0119] A determination unit 302, configured to determine a target area containing a light source in the image to be encoded;

[0120] A training unit 303 is used to train an initial imaging model using all pixels in the target area in the image to be encoded as samples to obtain a target imaging model, wherein the target imaging model is used to generate pixel values ​​according to input pixel coordinates;

[0121] The simulation unit 304 is used to input the pixel coordinates of the target area into the target imaging model to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the image to be encoded;

[0122] A generating unit 305, configured to remove the simulated light source image from the image to be encoded to obtain a first image;

[0123] The encoding unit 306 is used to encode the first image, the indication information indicating the target area and the model parameters of the target imaging model respectively to obtain the second image, the encoded indication information and the encoded model parameters.

[0124] The determination unit 302 is specifically configured to:

[0125] Determining the light source from the image to be encoded;

[0126] A rectangular region of a target area centered on the light source is determined as the target region.

[0127] The indication information is the coordinate value of the light source.

[0128] The training unit 303 is specifically used for:

[0129] Input each pixel point in the target area of ​​the image to be encoded into the initial imaging model to obtain a predicted value of each pixel point;

[0130] The difference between the pixel value of each pixel in the image to be encoded and the predicted value of each pixel is used as the error of each pixel;

[0131] Sum the errors of all pixels in the target area to obtain the total difference;

[0132] The initial imaging model is updated according to the above total difference to obtain the target imaging model.

[0133] The generating unit 305 is specifically used for:

[0134] The pixel value of the first pixel point in the above-mentioned first image is determined as the difference between the pixel value of the above-mentioned first pixel point in the above-mentioned image to be encoded and the pixel value of the pixel point corresponding to the above-mentioned first pixel point in the above-mentioned simulated light source image, and the above-mentioned first pixel point is a pixel point in the above-mentioned target area.

[0135] The generating unit 305 is further used for:

[0136] Determine the difference between the pixel value of a second pixel point in the image to be encoded and the pixel value of a pixel point corresponding to the second pixel point in the simulated light source image as a first difference value, wherein the second pixel point is any pixel point in the target area;

[0137] When the first difference is smaller than the target value, the second pixel is determined as the first pixel.

[0138] The units or modules described above as separate components may or may not be physically separated. The components described as units or modules may or may not be physical units, that is, they may be located in one device or distributed on multiple devices. The solutions of the embodiments in this disclosure may be implemented by selecting some or all of the units therein according to actual needs.

[0139] like Figure 5 As shown, Figure 5 4 is a schematic diagram of the structure of a decoder 40 provided in an embodiment of the present application. The decoder 40 may at least include:

[0140] A decoding unit 401 is used to decode the second image, the encoded indication information and the encoded model parameters respectively to obtain the first image, the indication information for indicating the target area and the model parameters;

[0141] An updating unit 402 is used to update the initial imaging model according to the above model parameters to obtain a target imaging model;

[0142] The acquisition unit 403 is used to acquire the pixel coordinates of the target area according to the indication information;

[0143] The simulation unit 404 is used to input the pixel coordinates of the target area into the target imaging model to obtain a simulated light source image, and the pixel points in the simulated light source image correspond to the pixel points in the target area in the image to be encoded.

[0144] The compensation unit 405 is used to compensate the first image according to the simulated light source image to obtain an original image.

[0145] The compensation unit 405 is specifically used for:

[0146] The pixel value of the first pixel point in the above-mentioned original image is determined as the sum of the pixel value of the above-mentioned first pixel point in the above-mentioned first image and the pixel value of the pixel point corresponding to the above-mentioned first pixel point in the above-mentioned simulated light source image, and the above-mentioned first pixel point is a pixel point in the above-mentioned target area.

[0147] The compensation unit 405 is further used for:

[0148] When the pixel value of the second pixel point in the first image is less than the target value, the second pixel point is determined as the first pixel point, and the second pixel point is any pixel point in the target area.

[0149] The units or modules described above as separate components may or may not be physically separated. The components described as units or modules may or may not be physical units, that is, they may be located in one device or distributed on multiple devices. The solutions of the embodiments in this disclosure may be implemented by selecting some or all of the units therein according to actual needs.

[0150] like Figure 6 As shown, Figure 6 It is a structural diagram of an encoder provided in an embodiment of the present application. The encoder may include: a processor 501, a memory 502, a communication bus 503 and a communication interface 504. The processor 501 is connected to the memory 502 and the communication interface 503 via the communication bus.

[0151] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 501 may be any conventional processor, etc.

[0152] The processor 501 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image encoding method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 501. The above-mentioned processor 501 may also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and executes the encoding method of the embodiment of the method of the present application in combination with its hardware.

[0153] The memory 502 may be a read-only memory (ROM), a random access memory (RAM) or other memory. In the embodiment of the present application, the memory 502 is used to store data and various software programs, such as a program for implementing the encoding method in the embodiment of the present application according to the function position relationship in the embodiment of the present application.

[0154] Optionally, in the embodiment of the present application, the above-mentioned memory may include a physical device for storing information, which is usually to digitize the information and then store it in a medium using electrical, magnetic or optical methods. The above-mentioned memory in this embodiment may also include: a device for storing information using electrical energy, such as RAM, ROM, etc.; a device for storing information using magnetic energy, such as a hard disk, a floppy disk, a magnetic tape, a magnetic core memory, a magnetic bubble memory, a USB flash drive; a device for storing information using optical means, such as a CD or a DVD. Of course, there are other types of memory, such as quantum memory, graphene memory, etc.

[0155] The communication interface 504 uses a transceiver such as, but not limited to, a transceiver to implement communication between the encoding device 50 and other devices or a communication network. For example, the neural network model can be obtained through the communication interface 504.

[0156] Optionally, the encoder 50 may further include an artificial intelligence processor 505. The artificial intelligence processor 505 may be mounted on the host CPU as a coprocessor, and the host CPU may assign tasks to it. The artificial intelligence processor 505 may implement one or more operations involved in the above encoding method. For example, taking a neural network processor (network processing unit, NPU) NPU as an example, the core part of the NPU is an operation circuit, which is controlled by a controller to extract matrix data in the memory 502 and perform multiplication and addition operations.

[0157] In the embodiment of the present application, the above-mentioned artificial intelligence processor 505 is also referred to as a dedicated processor, a processor for a specific application or field. For example: Graphics Processing Unit (GPU), also known as display core, visual processor, display chip, is a dedicated processor for image computing on personal computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.). Another example: Neural Processing Unit (NPU), is a dedicated processor for matrix multiplication operations in applications in the field of artificial intelligence. It adopts the architecture of "data-driven parallel computing" and is particularly good at processing massive multimedia data such as videos and images.

[0158] Optionally, the artificial intelligence processor can be an artificial intelligence processor with a reconfigurable architecture. Here, reconfigurable architecture means that if an artificial intelligence processor can use reusable hardware resources to flexibly change its own architecture according to different application requirements so as to provide an architecture that matches each specific application requirement, then this artificial intelligence processor is called a reconfigurable computing system, and its architecture is called a reconfigurable architecture.

[0159] The processor 501 is used to call the data and program code in the memory 502 to execute:

[0160] Obtaining an image to be encoded;

[0161] Determining a target area containing a light source in the image to be encoded;

[0162] Using all the pixels in the target area in the image to be encoded as samples to train an initial imaging model, a target imaging model is obtained, and the target imaging model is used to generate pixel values ​​according to the input pixel coordinates;

[0163] Inputting the pixel coordinates of the target area into the target imaging model to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the image to be encoded;

[0164] Eliminating the simulated light source image from the image to be encoded to obtain a first image;

[0165] The first image, the indication information for indicating the target area, and the model parameters of the target imaging model are encoded respectively to obtain a second image, the encoded indication information, and the encoded model parameters.

[0166] In one possible implementation manner, the processor determines a target area containing a light source in the image to be encoded, including:

[0167] Determining the light source from the image to be encoded;

[0168] A rectangular region of a target area centered on the light source is determined as the target region.

[0169] In one possible implementation manner, the indication information is a coordinate value of the light source.

[0170] In one possible implementation, the processor uses all pixel points in the target area in the image to be encoded as samples to train an initial imaging model to obtain a target imaging model, including:

[0171] Input each pixel point in the target area of ​​the image to be encoded into the initial imaging model to obtain a predicted value of each pixel point;

[0172] The difference between the pixel value of each pixel in the to-be-encoded image and the predicted value of each pixel is taken as the error of each pixel;

[0173] Sum the errors of all pixels in the target area to obtain the total difference;

[0174] The initial imaging model is updated according to the above total difference to obtain the target imaging model.

[0175] In one possible implementation manner, the processor removes the simulated light source image from the image to be encoded to obtain a first image, including:

[0176] The pixel value of the first pixel point in the above-mentioned first image is determined as the difference between the pixel value of the above-mentioned first pixel point in the above-mentioned image to be encoded and the pixel value of the pixel point corresponding to the above-mentioned first pixel point in the above-mentioned simulated light source image, and the above-mentioned first pixel point is a pixel point in the above-mentioned target area.

[0177] In one possible implementation manner, the processor executing the above method further includes:

[0178] Determine the difference between the pixel value of a second pixel point in the image to be encoded and the pixel value of a pixel point corresponding to the second pixel point in the simulated light source image as a first difference value, wherein the second pixel point is any pixel point in the target area;

[0179] When the first difference is smaller than the target value, the second pixel is determined as the first pixel.

[0180] The specific functions implemented by its memory and processor can be explained in comparison with the aforementioned implementation modes in this specification, and can achieve the technical effects of the aforementioned implementation modes, so they will not be repeated here.

[0181] like Figure 7 As shown, Figure 7 It is a structural diagram of a decoder provided in an embodiment of the present application. The above decoder may include: a processor 601, a memory 602, a communication bus 603 and a communication interface 604. The processor 601 is connected to the above memory 602 and the above communication interface 603 via the above communication bus.

[0182] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 601 may be any conventional processor, etc.

[0183] The processor 601 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image decoding method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 601. The above-mentioned processor 601 may also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and executes the image decoding method of the embodiment of the method of the present application in combination with its hardware.

[0184] The memory 602 may be a read-only memory (ROM), a random access memory (RAM) or other memory. In the embodiment of the present application, the memory 602 is used to store data and various software programs, such as a program for implementing the image decoding method in the embodiment of the present application according to the function position relationship in the embodiment of the present application.

[0185] Optionally, in the embodiment of the present application, the above-mentioned memory may include a physical device for storing information, which is usually to digitize the information and then store it in a medium using electrical, magnetic or optical methods. The above-mentioned memory in this embodiment may also include: a device for storing information using electrical energy, such as RAM, ROM, etc.; a device for storing information using magnetic energy, such as a hard disk, a floppy disk, a magnetic tape, a magnetic core memory, a magnetic bubble memory, a USB flash drive; a device for storing information using optical means, such as a CD or a DVD. Of course, there are other types of memory, such as quantum memory, graphene memory, etc.

[0186] The communication interface 604 uses a transceiver such as, but not limited to, a transceiver to implement communication between the image decoding device 60 and other devices or a communication network. For example, a neural network model can be obtained through the communication interface 604.

[0187] Optionally, the decoder 60 may further include an artificial intelligence processor 605. The artificial intelligence processor 605 may be mounted on the host CPU as a coprocessor, and the host CPU may assign tasks to it. The artificial intelligence processor 605 may implement one or more operations involved in the above-mentioned image decoding method. For example, taking a neural network processor (network processing unit, NPU) NPU as an example, the core part of the NPU is an operation circuit, which is controlled by a controller to extract matrix data in the memory 602 and perform multiplication and addition operations.

[0188] In the embodiment of the present application, the artificial intelligence processor 605 is also referred to as a dedicated processor, a processor for a specific application or field. For example: a graphics processing unit (Graphics Processing Unit, abbreviated as: GPU), also known as a display core, a visual processor, a display chip, is a dedicated processor for image computing on personal computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.). Another example: a neural network processor (Neural Processing Unit, abbreviated as: NPU), is a dedicated processor for matrix multiplication operations in applications in the field of artificial intelligence. It adopts a "data-driven parallel computing" architecture and is particularly good at processing massive multimedia data such as videos and images.

[0189] Optionally, the artificial intelligence processor can be an artificial intelligence processor with a reconfigurable architecture. Here, reconfigurable architecture means that if an artificial intelligence processor can use reusable hardware resources to flexibly change its own architecture according to different application requirements so as to provide an architecture that matches each specific application requirement, then this artificial intelligence processor is called a reconfigurable computing system, and its architecture is called a reconfigurable architecture.

[0190] The processor 601 is used to call the data and program code in the memory 602 to execute:

[0191] Decoding the second image, the encoded indication information, and the encoded model parameters respectively to obtain the first image, the indication information for indicating the target area, and the model parameters;

[0192] Update the initial imaging model according to the above model parameters to obtain the target imaging model;

[0193] According to the above indication information, the pixel coordinates of the above target area are obtained;

[0194] Inputting the pixel coordinates of the target area into the target imaging model to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the image to be encoded;

[0195] The first image is compensated according to the simulated light source image to obtain an original image.

[0196] In a possible implementation, compensating the first image according to the simulated light source image to obtain the original image includes:

[0197] The pixel value of the first pixel point in the above-mentioned original image is determined as the sum of the pixel value of the above-mentioned first pixel point in the above-mentioned first image and the pixel value of the pixel point corresponding to the above-mentioned first pixel point in the above-mentioned simulated light source image, and the above-mentioned first pixel point is a pixel point in the above-mentioned target area.

[0198] In one possible implementation manner, the processor executing the above method further includes:

[0199] When the pixel value of the second pixel point in the first image is less than the target value, the second pixel point is determined as the first pixel point, and the second pixel point is any pixel point in the target area.

[0200] The present application also provides a computer storage medium for storing computer software instructions used by the above encoder and decoder, which includes a program for executing the above method embodiment. By executing the stored program, the optimization of the neural network model can be achieved to reduce redundant calculations.

[0201] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0202] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable encoder or decoder to produce a machine, so that the instructions executed by the processor of the computer or other programmable encoder or decoder generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0203] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable encoder or decoder to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0204] These computer program instructions may also be loaded onto a computer or other programmable encoder or decoder so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0205] The embodiments of the present disclosure are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present disclosure. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present disclosure, the specific implementation methods and the scope of application of the present disclosure, all belong to the scope of protection of the present disclosure. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. An image coding method, characterized in that: include: Obtaining an image to be encoded; Determining a target area containing a light source in the image to be encoded; Using all pixel points in the target area of ​​the image to be encoded as samples to train an initial imaging model to obtain a target imaging model, wherein the target imaging model is used to generate pixel values ​​according to input pixel point coordinates; Inputting the pixel coordinates of the target area into the target imaging model respectively to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the image to be encoded; Eliminating the simulated light source image from the image to be encoded to obtain a first image; respectively encoding the first image, the indication information indicating the position of the target area, and the model parameters of the target imaging model to obtain a second image, the encoded indication information, and the encoded model parameters; The step of removing the simulated light source image from the image to be encoded to obtain a first image includes: The pixel value of the first pixel point in the first image is determined as the difference between the pixel value of the first pixel point in the image to be encoded and the pixel value of the pixel point corresponding to the first pixel point in the simulated light source image, and the first pixel point is a pixel point in the target area.

2. The method according to claim 1, characterized in that The step of determining a target area containing a light source in the image to be encoded comprises: Determining the light source from the image to be encoded; A rectangular region of a target area centered on the light source is determined as the target region.

3. The method according to claim 2, characterized in that The indication information is the coordinate value of the light source.

4. The method according to claim 1, characterized in that The step of training the initial imaging model using all the pixels in the target area in the image to be encoded as samples to obtain the target imaging model includes: Inputting each pixel point in the target area of ​​the image to be encoded into the initial imaging model to obtain a predicted value of each pixel point; Taking the difference between the pixel value of each pixel in the to-be-encoded image and the predicted value of each pixel as the error of each pixel; Sum the errors of all pixels in the target area to obtain a total difference; The initial imaging model is updated according to the total difference to obtain a target imaging model.

5. The method according to claim 1, characterized in that The method further comprises: Determine a difference between a pixel value of a second pixel point in the image to be encoded and a pixel value of a pixel point corresponding to the second pixel point in the simulated light source image as a first difference value, wherein the second pixel point is any pixel point in the target area; When the first difference is smaller than a target value, the second pixel is determined as the first pixel.

6. An image decoding method, characterized in that: include: Decoding the second image, the encoded indication information, and the encoded model parameters respectively to obtain the first image, the indication information for indicating the position of the target area, and the model parameters; the second image, the encoded indication information, and the encoded model parameters are determined based on the method according to any one of claims 1 to 5; Update the initial imaging model according to the model parameters to obtain a target imaging model; According to the indication information, obtaining pixel coordinates of the target area; Inputting the pixel coordinates of the target area into the target imaging model respectively to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the first image; The first image is compensated according to the simulated light source image to obtain an original image.

7. The method according to claim 6, characterized in that The compensating the first image according to the simulated light source image to obtain an original image includes: The pixel value of the first pixel point in the original image is determined as the sum of the pixel value of the first pixel point in the first image and the pixel value of the pixel point corresponding to the first pixel point in the simulated light source image, and the first pixel point is a pixel point in the target area.

8. The method according to claim 7, characterized in that The method further comprises: When a pixel value of a second pixel point in the first image is less than a target value, the second pixel point is determined as the first pixel point, and the second pixel point is any pixel point in the target area.

9. An encoder, characterized in that: include: A first acquisition unit, used to acquire an image to be encoded; a determination unit, configured to determine a target area containing a light source in the image to be encoded; A training unit, used to train an initial imaging model using all pixel points in the target area in the image to be encoded as samples to obtain a target imaging model, wherein the target imaging model is used to generate pixel values ​​according to input pixel point coordinates; A simulation unit, used for inputting the pixel coordinates of the target area into the target imaging model respectively to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the image to be encoded; A generating unit, configured to remove the simulated light source image from the image to be encoded to obtain a first image; an encoding unit, used to encode the first image, the indication information indicating the position of the target area, and the model parameters of the target imaging model, respectively, to obtain a second image, the encoded indication information, and the encoded model parameters; The generating unit is specifically used for: The pixel value of the first pixel point in the first image is determined as the difference between the pixel value of the first pixel point in the image to be encoded and the pixel value of the pixel point corresponding to the first pixel point in the simulated light source image, and the first pixel point is a pixel point in the target area.

10. A decoder, characterized in that: include: a decoding unit, used to decode the second image, the encoded indication information and the encoded model parameters respectively to obtain the first image, the indication information for indicating the position of the target area and the model parameters; the second image, the encoded indication information and the encoded model parameters are obtained based on the encoder according to claim 9; An updating unit, used for updating the initial imaging model according to the model parameters to obtain a target imaging model; An acquisition unit, used for acquiring pixel coordinates of the target area according to the indication information; A simulation unit, used for inputting the pixel coordinates of the target area into the target imaging model respectively to obtain a simulated light source image, wherein the pixel points in the simulated light source image correspond one-to-one to the pixel points in the target area in the first image; A compensation unit is used to compensate the first image according to the simulated light source image to obtain an original image.

11. A computer device, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are connected to each other, wherein the processor comprises a general-purpose processor and an artificial intelligence processor, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 8.

Images