Image processing method and device, electronic device and computer readable storage medium

By using macroblock position and encoding information to determine the block effect area in image processing, the problems of large data processing volume and low efficiency in the prior art are solved, and more efficient block effect area recognition and image quality improvement are achieved.

CN117750037BActive Publication Date: 2024-11-19SHUXING TECH (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310241877.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-14
Publication Date
2024-11-19
Estimated Expiration
2043-03-14

AI Technical Summary

Technical Problem

The prior art has a large amount of data processing and low efficiency when determining the block effect area from an image, which affects the image quality.

Method used

By obtaining the target video frame and its encoding information, the macroblock boundary is determined using the macroblock position, the pixel neighborhood is constructed, the boundary pixel number and gradient are calculated, and the threshold value is combined to determine whether the pixel is a block effect pixel, thereby determining the block effect area , and smooth it.

Benefits of technology

The efficiency of block effect area determination is improved, the data processing volume is reduced, and the image quality is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117750037B_ABST
    Figure CN117750037B_ABST
Patent Text Reader

Abstract

The present application discloses an image processing method and device, an electronic device and a computer-readable storage medium. The method comprises: obtaining a target video frame and encoding information of the target video frame, wherein the encoding information comprises a macroblock position of a macroblock in the target video frame; determining a boundary position of a macroblock boundary in the target video frame based on the macroblock position; determining a block effect region from the target video frame based on the boundary position, wherein the block effect region is a region where a block effect occurs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Image compression may cause block effects in the image, which will obviously reduce the image quality. Therefore, in order to improve the image quality, it is necessary to eliminate the block effects in the image. Before eliminating the block effects in the image, it is necessary to determine the area where the block effects appear in the image. Therefore, how to determine the block effect area in the image is of great significance. Summary of the invention

[0003] The present application provides an image processing method and device, an electronic device, and a computer-readable storage medium.

[0004] In a first aspect, an image processing method is provided, the method comprising:

[0005] Acquire a target video frame and encoding information of the target video frame, wherein the encoding information includes a macroblock position of a macroblock in the target video frame;

[0006] Determine the boundary position of the boundary of the macroblock in the target video frame based on the macroblock position;

[0007] A blocking effect area is determined from the target video frame based on the boundary position, where the blocking effect area is an area where the blocking effect occurs.

[0008] In combination with any implementation manner of the present application, the determining the blocking effect area from the target video frame based on the boundary position includes:

[0009] Determining boundary pixels located on the boundary of the macroblock based on the boundary position;

[0010] Constructing a pixel neighborhood with a preset size with the pixel to be confirmed in the target video frame as the center;

[0011] determining a target number of said boundary pixels within said pixel neighborhood;

[0012] When the target number is greater than or equal to a first threshold, determining that the pixel to be confirmed is a blocking pixel;

[0013] The blocking effect area is determined according to the blocking effect pixels in the target video frame.

[0014] In combination with any implementation manner of the present application, when the target number is greater than or equal to the first threshold, before determining that the pixel to be confirmed is a blocking pixel, the method further includes:

[0015] Determine the average gradient of the boundary pixels in the pixel neighborhood to obtain a first gradient;

[0016] Determine the average gradient of pixels in the pixel neighborhood except the boundary pixels to obtain a second gradient;

[0017] The step of determining that the pixel to be confirmed is a blocking pixel when the target number is greater than or equal to a first threshold comprises:

[0018] When the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value, it is determined that the pixel to be confirmed is the blocking pixel, and the preset value is greater than 1.

[0019] In combination with any implementation manner of the present application, determining the average gradient of the boundary pixels in the pixel neighborhood to obtain the first gradient includes:

[0020] Determine the gradient sum of the boundary pixels in the pixel neighborhood by using a Sobel operator;

[0021] The quotient of the gradient sum of the boundary pixels and the target number is determined to obtain the first gradient.

[0022] In combination with any implementation manner of the present application, the determining the blocking effect area from the target video frame based on the boundary position includes:

[0023] Determine pixels in the target video frame whose distance to the boundary position is less than or equal to a second threshold as blocking pixels;

[0024] The blocking effect area is determined according to the blocking effect pixels in the target video frame.

[0025] In combination with any implementation manner of the present application, the step of determining that a pixel in the target video frame whose distance to the boundary position is less than or equal to a second threshold is a blocking pixel includes:

[0026] determining a flat area from the target video frame;

[0027] The pixels in the flat area whose distance to the boundary position is less than or equal to a second threshold are determined as the blocking pixels.

[0028] In combination with any implementation manner of the present application, after determining the blocking area from the target video frame based on the boundary position, the method further includes:

[0029] The block effect area in the target video frame is smoothed to obtain an enhanced video frame.

[0030] In combination with any implementation manner of the present application, the step of smoothing the block effect area in the target video frame to obtain an enhanced video frame includes:

[0031] The area other than the block effect area in the target video frame is sharpened, and the block effect area in the target video frame is smoothed to obtain the enhanced video frame.

[0032] In a second aspect, an image processing device is provided, the device comprising:

[0033] An acquisition unit, configured to acquire a target video frame and encoding information of the target video frame, wherein the encoding information includes a macroblock position of a macroblock in the target video frame;

[0034] A determination unit, configured to determine a boundary position of a macroblock boundary in the target video frame based on the macroblock position;

[0035] The determining unit is further configured to determine a blocking effect region from the target video frame based on the boundary position, wherein the blocking effect region is a region where blocking effects occur.

[0036] In combination with any implementation manner of the present application, the determining unit is used to:

[0037] Determining boundary pixels located on the boundary of the macroblock based on the boundary position;

[0038] Constructing a pixel neighborhood with a preset size with the pixel to be confirmed in the target video frame as the center;

[0039] determining a target number of said boundary pixels within said pixel neighborhood;

[0040] When the target number is greater than or equal to a first threshold, determining that the pixel to be confirmed is a blocking pixel;

[0041] The blocking effect area is determined according to the blocking effect pixels in the target video frame.

[0042] In combination with any implementation manner of the present application, the determining unit is further configured to:

[0043] Determine the average gradient of the boundary pixels in the pixel neighborhood to obtain a first gradient;

[0044] Determine the average gradient of pixels in the pixel neighborhood except the boundary pixels to obtain a second gradient;

[0045] When the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value, it is determined that the pixel to be confirmed is the blocking pixel, and the preset value is greater than 1.

[0046] In combination with any implementation manner of the present application, the determining unit is used to:

[0047] Determine the gradient sum of the boundary pixels in the pixel neighborhood by using a Sobel operator;

[0048] The quotient of the gradient sum of the boundary pixels and the target number is determined to obtain the first gradient.

[0049] In combination with any implementation manner of the present application, the determining unit is used to:

[0050] Determine pixels in the target video frame whose distance to the boundary position is less than or equal to a second threshold as blocking pixels;

[0051] The blocking effect area is determined according to the blocking effect pixels in the target video frame.

[0052] In combination with any implementation manner of the present application, the determining unit is used to:

[0053] determining a flat area from the target video frame;

[0054] The pixels in the flat area whose distance to the boundary position is less than or equal to a second threshold are determined as the blocking pixels.

[0055] In combination with any embodiment of the present application, the device further includes:

[0056] The processing unit is used to perform smoothing on the block effect area in the target video frame to obtain an enhanced video frame.

[0057] In combination with any embodiment of the present application, the processing unit is used to:

[0058] The area other than the block effect area in the target video frame is sharpened, and the block effect area in the target video frame is smoothed to obtain the enhanced video frame.

[0059] According to a third aspect, an electronic device is provided, comprising: a processor and a memory, wherein the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to the first aspect and any possible implementation thereof.

[0060] In a fourth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the first aspect and any one of its embodiments.

[0061] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the first aspect and any one of its embodiments.

[0062] In a sixth aspect, a computer program product is provided, wherein the computer program product comprises a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the above-mentioned first aspect and any embodiment thereof.

[0063] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0064] In the present application, after acquiring the target video frame and the coding information of the target video frame, the image processing device uses the macroblock position in the coding information to determine the boundary position of the macroblock boundary in the target video frame. Since the block effect is usually easy to appear at the boundary of the macroblock, and the closer the area is to the boundary of the macroblock, the higher the probability of the block effect, the image processing device can determine the block effect area from the target video frame based on the boundary position, thereby determining the block effect area in the target video frame based on the coding information of the target video frame.

[0065] Since the encoding information of the target video frame is the information carried by the target video frame, determining the blocking effect area in the target video frame based on the encoding information of the target video frame can improve the efficiency of determining the blocking effect area in the target video frame and reduce the data processing amount of determining the blocking effect area in the target video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0067] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0068] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0069] Figure 2 A schematic diagram of a target video frame provided in an embodiment of the present application;

[0070] Figure 3 A flowchart of another image processing method provided in an embodiment of the present application;

[0071] Figure 4 A method provided in an embodiment of the present application is to determine Figure 2 A schematic diagram of a boundary map E obtained by the boundary position in the target video frame shown;

[0072] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0076] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0077] Compressing an image may cause a block effect to appear in the image. For example, encoding a video may cause a block effect to appear in the video frame of the video. The appearance of the block effect will obviously reduce the image quality. Therefore, in order to improve the image quality, the block effect in the image needs to be eliminated. Before eliminating the block effect in the image, it is necessary to determine the area where the block effect appears (hereinafter referred to as the block effect area) from the image. The current technology usually determines the block effect area from the image through a specific block effect area detection algorithm, which results in a large amount of data processing and low efficiency for determining the block effect area from the image. Based on this, an embodiment of the present application provides an image processing method to reduce the amount of data processing for determining the block effect area from the image and improve efficiency.

[0078] The execution subject of the embodiment of the present application is an image processing device, wherein the image processing device can be any electronic device that can execute the technical solution disclosed in the embodiment of the method of the present application. Optionally, the image processing device can be one of the following: a mobile phone, a computer, a tablet computer, and a wearable smart device.

[0079] It should be understood that the method embodiment of the present application can also be implemented by a processor executing a computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 It is a flowchart of an image processing method provided in an embodiment of the present application.

[0080] 101. Obtain a target video frame and encoding information of the target video frame.

[0081] In the embodiment of the present application, the target video frame is any frame in the encoded video obtained by encoding the video to be encoded. The target video frame is obtained by encoding the video frame to be encoded in the video to be encoded. For example, in the process of encoding video A to obtain video B, the first frame in video A is encoded to obtain the first frame in video B. At this time, video A is the video to be encoded, and video B is the encoded video. If the first frame in video A is the video frame to be encoded, then the first frame in video B is the target video frame.

[0082] In the embodiment of the present application, the encoding information of the target video includes information for encoding the video frame to be encoded, wherein the encoding information includes the macroblock position of the macroblock in the target video frame, and the macroblock position represents the position of the macroblock in the target video frame. The macroblock is a coding unit for video encoding. For example, the target video frame is obtained by encoding the video frame to be encoded. Then, in the process of encoding the video frame to be encoded to obtain the target video frame, the video frame to be encoded is divided into at least two macroblocks, and then each macroblock is encoded separately to obtain the target video frame. For example, Figure 2 The target video frame is shown in Figure 2Some macroblocks have been visualized in , specifically, Figure 2 There is an orange square on the back of the hand in the lower middle part, which includes multiple small squares, each of which is a macroblock.

[0083] In an implementation of acquiring a target video frame, an image processing device receives a target video frame input by a user through an input component to acquire the target video frame. The input component includes: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0084] In another implementation of obtaining a target video frame, the image processing device receives a target video frame sent by a terminal to obtain the target video frame. Optionally, the terminal may be any one of the following: a mobile phone, a computer, a tablet computer, and a server. For example, the terminal is a server, and a communication connection exists between the server and the image processing device. The server sends the encoded video to the image processing device through the communication connection, and the image processing device obtains the target video frame by receiving the encoded video.

[0085] In an implementation method of acquiring encoding information of a target video frame, an image processing apparatus receives a target video frame input by a user through an input component to acquire the target video frame.

[0086] In another implementation of obtaining the encoding information of the target video frame, the image processing device receives the target video frame sent by the terminal to obtain the target video frame. For example, the terminal is a server, and there is a communication connection between the server and the image processing device. The server sends the encoded video to the image processing device through the communication connection. At this time, the encoded video carries the encoding information of the encoded video, and the image processing device uses the encoding information of the encoded video as the encoding information of the target video frame.

[0087] 102. Determine a boundary position of a macroblock boundary in the target video frame based on the macroblock position.

[0088] In the embodiment of the present application, the boundary position is the position of the boundary of the macroblock in the target video frame. The image processing device can determine the position of the boundary of the macroblock in the target video frame according to the macroblock position, thereby determining the boundary position.

[0089] 103. Determine a blocking effect area from the target video frame based on the boundary position.

[0090] In the embodiment of the present application, the blocking effect area is an area where the blocking effect appears in the target video frame, for example, Figure 2 In the target video frame shown, there are block effects on the back of the hand. Since block effects are usually easy to appear at the boundary of macroblocks, and the closer the area is to the boundary of the macroblock, the higher the probability of block effects appearing, the image processing device can determine the block effect area from the target video frame based on the boundary position.

[0091] In the embodiment of the present application, after acquiring the target video frame and the coding information of the target video frame, the image processing device uses the macroblock position in the coding information to determine the boundary position of the macroblock boundary in the target video frame. Because the blocking effect is usually easy to appear at the boundary of the macroblock, and the closer the area to the boundary of the macroblock is, the higher the probability of the blocking effect, the image processing device can determine the blocking effect area from the target video frame based on the boundary position, thereby determining the blocking effect area in the target video frame based on the coding information of the target video frame.

[0092] Since the encoding information of the target video frame is the information carried by the target video frame, determining the blocking effect area in the target video frame based on the encoding information of the target video frame can improve the efficiency of determining the blocking effect area in the target video frame and reduce the data processing amount of determining the blocking effect area in the target video frame.

[0093] As an optional implementation manner, the image processing apparatus performs the following steps during the process of executing step 103:

[0094] 201. Determine boundary pixels located on the boundary of the macroblock based on the boundary position.

[0095] In the embodiment of the present application, pixels located at the boundary of a macroblock are referred to as boundary pixels. The image processing apparatus may use pixels located at boundary positions in the target video frame as boundary pixels.

[0096] 202. Construct a pixel neighborhood with a preset size with the to-be-confirmed pixel in the target video frame as the center.

[0097] In the embodiment of the present application, the pixel to be confirmed can be any pixel in the target video frame. The preset size is a pre-set size. Optionally, the preset size is N×N, where N is an odd number. For example, the preset size is 5×5, and the size of the pixel neighborhood is 5×5. For another example, the preset size is 7×7, and the size of the pixel neighborhood is 7×7. The center of the pixel neighborhood is the pixel to be confirmed, that is, the pixel to be confirmed is the geometric center of the pixel neighborhood.

[0098] 203. Determine a target number of the boundary pixels within the pixel neighborhood.

[0099] In the embodiment of the present application, the target number is the number of boundary pixels in the pixel neighborhood.

[0100] 204. When the target number is greater than or equal to a first threshold, determine that the pixel to be confirmed is a blocking pixel.

[0101] Since the size of the pixel neighborhood is a preset size, the size of the pixel neighborhood is fixed, that is, the number of pixels in the pixel neighborhood is fixed. Therefore, if the number of boundary pixels in the pixel neighborhood is large, it means that the center of the pixel neighborhood is close to the macroblock boundary, which means that the probability of block effect at the position where the pixel to be confirmed is located is high. Conversely, if the number of boundary pixels in the pixel neighborhood is small, it means that the center of the pixel neighborhood is far from the macroblock boundary, which means that the probability of block effect at the position where the pixel to be confirmed is located is low.

[0102] In an embodiment of the present application, the image processing device determines whether the number of boundary pixels in a pixel neighborhood is large or small based on a first threshold value. Specifically, if the number of boundary pixels in a pixel neighborhood is greater than or equal to the first threshold value, it means that the number of boundaries in the pixel neighborhood is large; if the number of boundary pixels in a pixel neighborhood is less than the first threshold value, it means that the number of boundaries in the pixel neighborhood is small.

[0103] Therefore, when the target number is greater than or equal to the first threshold, the image processing device determines that the probability of the block effect occurring at the position where the pixel to be confirmed is located is high, and further determines that the pixel to be confirmed is a block effect pixel. Conversely, when the target number is less than the first threshold, the image processing device determines that the probability of the block effect occurring at the position where the pixel to be confirmed is located is low, and further determines that the pixel to be confirmed is not a block effect pixel. In the embodiment of the present application, the block effect pixel is a pixel with a high probability of the block effect occurring.

[0104] 205. Determine the blocking effect area according to the blocking effect pixels in the target video frame.

[0105] In a possible implementation, the image processing apparatus uses a pixel region including blocking effect pixels as a blocking effect region. Optionally, the image processing apparatus uses a pixel region composed of blocking effect pixels as a blocking effect region.

[0106] In this embodiment, the image processing device determines the boundary pixels located on the boundary of the macroblock based on the boundary position, and constructs a pixel neighborhood with a preset size and a preset size with the pixel to be confirmed in the target video frame as the center, and then determines the target number of boundary pixels in the pixel neighborhood. Then, based on the size relationship between the target number and the first threshold, it is determined whether the pixel to be confirmed is a blocking pixel. Specifically, when the target number is greater than or equal to the first threshold, the pixel to be confirmed is determined to be a blocking pixel. That is, the image processing device can determine whether each pixel in the target video frame is a blocking pixel based on steps 201 to 204, and after determining all the blocking pixels in the target video frame, the blocking area can be determined according to the blocking pixels in the target video frame.

[0107] As an optional implementation manner, before executing step 204, the image processing device further executes the following steps:

[0108] 301. Determine an average gradient of the boundary pixels in the pixel neighborhood to obtain a first gradient.

[0109] In a possible implementation manner, the image processing device may determine the gradient of each boundary pixel in the pixel neighborhood respectively, and then determine the average value of the gradients of the boundary pixels in the pixel neighborhood to obtain the first gradient.

[0110] In another possible implementation, the image processing device determines the gradient sum of the boundary pixels in the pixel neighborhood by using a Sobel operator, and then determines the quotient of the gradient sum of the boundary pixels and the target number to obtain a first gradient.

[0111] 302. Determine an average gradient of pixels in the pixel neighborhood except the boundary pixels to obtain a second gradient.

[0112] For the sake of convenience, pixels other than boundary pixels in a pixel neighborhood are referred to as non-boundary pixels below. In one possible implementation, the image processing device may separately determine the gradient of each non-boundary pixel in the pixel neighborhood, and then determine the average value of the gradients of the non-boundary pixels in the pixel neighborhood to obtain a second gradient.

[0113] In another possible implementation, the image processing device determines the gradient sum of non-boundary pixels in the pixel neighborhood by using a Sobel operator, determines the number of non-boundary pixels in the pixel neighborhood, and then determines the quotient of the gradient sum of the non-boundary pixels and the number of non-boundary pixels to obtain a second gradient.

[0114] When the first gradient and the second gradient are obtained, the image processing apparatus performs the following steps during the process of executing step 204:

[0115] 303 . When the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value, determine that the pixel to be confirmed is the blocking pixel.

[0116] There may be a region with a large gradient in the target video frame (hereinafter referred to as a large gradient region), and when the blocking effect appears in the large gradient region, the blocking effect has little effect on the display effect of the target video frame. Specifically, when the blocking effect appears in the large gradient region, it is usually difficult for the naked eye to detect the existence of the blocking effect. Therefore, instead of detecting the blocking effect region in the large gradient region, the blocking effect region in the region other than the large gradient region (hereinafter referred to as a flat region) may be detected. In this way, the display effect of the target video frame is improved by removing the blocking effect in the blocking effect region in the flat region, that is, the image quality of the target video frame is improved, and the amount of data processing for improving the image quality of the target video frame can be reduced.

[0117] Because the gradient of non-boundary pixels in a large gradient area may be large, while the gradient of non-boundary pixels in a flat area is usually small, when the gradient of the boundary pixel is large, the difference between the average gradient of the boundary pixels in the pixel neighborhood and the average gradient of the non-boundary pixels in the pixel neighborhood can be used as a basis to determine whether the pixel neighborhood is a large gradient area or a flat area. Specifically, the difference between the average gradient of the boundary pixels in the pixel neighborhood and the average gradient of the non-boundary pixels in the pixel neighborhood is large, that is, the difference between the first gradient and the second gradient is large, indicating that the pixel neighborhood is a flat area, and the difference between the average gradient of the boundary pixels in the pixel neighborhood and the average gradient of the non-boundary pixels in the pixel neighborhood is small, that is, the difference between the first gradient and the second gradient is small, indicating that the pixel neighborhood is a large gradient area.

[0118] In the embodiment of the present application, the image processing device determines that the difference between the first gradient and the second gradient is large by determining whether the first gradient is greater than the product of the second gradient and a preset value, wherein the preset value is greater than 1. Specifically, the image processing device determines that the difference between the first gradient and the second gradient is large when the first gradient is greater than the product of the second gradient and the preset value, and the image processing device determines that the difference between the first gradient and the second gradient is small when the first gradient is less than or equal to the product of the second gradient and the preset value. Therefore, when the first gradient is greater than the product of the second gradient and the preset value, the image processing device determines that the pixel neighborhood is a flat area, that is, the pixel to be confirmed is a pixel in the flat area, and further determines whether the pixel to be confirmed is a block effect pixel. When the first gradient is less than or equal to the product of the second gradient and the preset value, the image processing device determines that the pixel neighborhood is a large gradient area, that is, the pixel to be confirmed is a pixel in the large gradient area, and further determines whether the pixel to be confirmed is a block effect pixel.

[0119] Therefore, the image processing apparatus determines that the pixel to be confirmed is a blocking pixel when the target number is greater than or equal to the first threshold and the first gradient is greater than the product of the second gradient and the preset value.

[0120] In this embodiment, before determining whether the pixel to be confirmed is a blocking pixel, the image processing device determines the average gradient of the boundary pixels in the pixel neighborhood to obtain the first gradient, and determines the average gradient of the non-boundary pixels in the pixel neighborhood to obtain the second gradient. Then, according to the difference between the first gradient and the second gradient, it is determined whether the pixel neighborhood is a large gradient area or a flat area, and when the pixel neighborhood is a flat area, the pixel to be confirmed is determined to be a pixel in the flat area, and then it is determined whether the pixel to be confirmed is a blocking pixel. Therefore, when the target number of the image is greater than or equal to the first threshold value, and the first gradient is greater than the product of the second gradient and the preset value, the pixel to be confirmed is determined to be a blocking pixel, and the blocking pixel can be a pixel with a high probability of appearing blocking in the flat area.

[0121] As an optional implementation manner, the image processing apparatus performs the following steps during the process of executing step 103:

[0122] 401 : Determine pixels in the target video frame whose distance to the boundary position is less than or equal to a second threshold as blocking pixels.

[0123] As described in step 103, blocking effects are usually prone to occur at the boundaries of macroblocks, and the closer the area is to the boundary of the macroblock, the higher the probability of the blocking effect occurring. Therefore, the image processing device can select pixels close to the boundary of the macroblock from the target video frame as blocking effect pixels. In other words, pixels close to the boundary position in the target video frame can be used as blocking effect pixels.

[0124] In the embodiment of the present application, the image processing device determines whether the distance from the pixel in the target video frame to the boundary position is close or far based on the second threshold value. Specifically, if the distance from the pixel in the target video frame to the boundary position is less than or equal to the second threshold value, it means that the distance from the pixel to the boundary position is close, and thus the pixel can be determined to be a blocking pixel. Conversely, if the distance from the pixel in the target video frame to the boundary position is greater than the second threshold value, it means that the distance from the pixel to the boundary position is far, and thus the pixel can be determined to be not a blocking pixel. Therefore, the image processing device determines that the pixel in the target video frame whose distance from the boundary position is less than or equal to the second threshold value is a blocking pixel.

[0125] 402. Determine the blocking effect area according to the blocking effect pixels in the target video frame.

[0126] The implementation of this step can refer to the implementation of step 205, which will not be repeated here.

[0127] In this embodiment, the image processing device determines whether the pixel in the target video frame is a blocking pixel based on the size relationship between the distance from the pixel in the target video frame to the boundary position and the second threshold value. Specifically, when the distance from the pixel in the target video frame to the boundary position is less than or equal to the second threshold value, the pixel in the target video frame is determined to be a blocking pixel. That is, the image processing device can determine whether each pixel in the target video frame is a blocking pixel based on step 401, and after determining all the blocking pixels in the target video frame, the blocking area can be determined according to the blocking pixels in the target video frame.

[0128] As an optional implementation manner, the image processing apparatus performs the following steps during the process of executing step 401:

[0129] 501. Determine a flat area from the target video frame.

[0130] In the embodiment of the present application, the flat area is an area with a small gradient, that is, an area other than the large gradient area described above. In one possible implementation, the average gradient of the pixels in the flat area is less than or equal to the third threshold. The average gradient of the pixels in the flat area is less than or equal to the third threshold, indicating that the average gradient of the pixels in the flat area is small, that is, the flat area is an area with a small gradient.

[0131] In another possible implementation, the flat region is a region in the target video frame where the grayscale change is less than or equal to the fourth threshold. The grayscale change in the flat region is less than or equal to the fourth threshold, indicating that the flat region is a region with a small gradient.

[0132] 502. Determine pixels in the flat area whose distance to the boundary position is less than or equal to a second threshold as the blocking pixel.

[0133] As described in step 303, when the blocking effect appears in the large gradient region, it is usually difficult for the naked eye to detect the existence of the blocking effect. Therefore, instead of detecting the blocking effect region in the large gradient region, the blocking effect region in the region other than the large gradient region (hereinafter referred to as the region other than the large gradient region as the flat region) may be detected. In this way, the display effect of the target video frame is improved by removing the blocking effect in the blocking effect region in the flat region, that is, the image quality of the target video frame is improved, and the amount of data processing for improving the image quality of the target video frame can be reduced. Therefore, in step 502, the image processing device determines the blocking effect pixels from the flat region in the target video frame. Specifically, the image processing device determines the pixels in the flat region whose distance to the boundary position is less than or equal to the second threshold as the blocking effect pixels.

[0134] In this embodiment, before determining whether the pixel to be confirmed is a blocking pixel, the image processing device determines a flat area from the target video frame, and then determines whether the pixel in the flat area is a blocking pixel. Specifically, the pixel in the flat area whose distance to the boundary position is less than or equal to the second threshold is determined to be a blocking pixel, so that the blocking pixel can be a pixel with a high probability of blocking in the flat area.

[0135] As an optional implementation manner, after determining the blocking area from the target video frame based on the boundary position, the image processing device further performs the following steps:

[0136] 601. Perform smoothing processing on the blocking area in the target video frame to obtain an enhanced video frame.

[0137] Because the blocking effect area in the target video frame may have a blocking effect, which leads to a decrease in the image quality of the target video frame, and the blocking effect in the blocking effect area can be eliminated by smoothing the blocking effect area in the target video frame, so the image processing device can eliminate the blocking effect in the blocking effect area by smoothing the blocking effect area in the target video frame, thereby improving the image quality of the target video frame.

[0138] As an optional implementation manner, the image processing apparatus performs the following steps during the process of executing step 601:

[0139] 701. Perform sharpening processing on the area except the blocking effect area in the target video frame, and perform smoothing processing on the blocking effect area in the target video frame to obtain the enhanced video frame.

[0140] In step 701, the image processing apparatus performs a sharpening process on the target video frame except the block effect region while performing a smoothing process on the block effect region in the target video frame, so that the block effect in the target video frame can be eliminated and the edge in the target video frame can be sharpened, thereby improving the image quality of the target video frame and sharpening the edge in the target video frame. This can avoid the situation where the boundary of the block effect region is also sharpened due to the sharpening process on all pixels in the target video frame.

[0141] In a possible implementation, the image processing device performs sharpening processing on the target video frame by using the following formula, so that while smoothing the block effect area in the target video frame, the area other than the block effect area in the target video frame is sharpened to obtain an enhanced video frame:

[0142] A pro =A+gain×(Af(A))…Formula (1)

[0143] Among them, A prorepresents the enhanced video frame, A represents the target video frame. f(A) represents the intermediate frequency signal and the low frequency signal in the target video frame. Optionally, f(·) represents Gaussian filtering, that is, f(A) represents the intermediate frequency signal in the target video frame and the low frequency signal in the target video frame obtained by performing Gaussian filtering on the target video frame and removing the high frequency signal in the target video frame.

[0144] gain represents gain. Optionally, for the block effect area in the target video frame, the value of gain is -1. pro =f(A), that is, in the process of obtaining an enhanced video frame by sharpening the target video frame, the high-frequency signal in the block effect area in the target video frame is removed, and the low-frequency signal and the medium-frequency signal in the block effect area in the target video frame are retained. In other words, the block effect area in the target video frame is smoothed. For the area other than the block effect area in the target video frame, the value of gain is a positive number.

[0145] Based on the technical solution provided in the embodiment of the present application, the embodiment of the present application also provides a possible application scenario. Figure 3 , Figure 3 FIG. 1 is a flow chart of another image processing method provided in an embodiment of the present application.

[0146] like Figure 3 As shown, after a code stream is input to an image processing device, the image processing device performs video decoding on the code stream to obtain an encoded video, wherein the encoded video includes a decoded video frame A. The decoded video frame A is used as the target video frame, and the encoding information of the target video frame can be obtained according to the encoding information of the encoded video. Then, according to the macroblock position in the encoding information of the target video frame, the possible position of the block effect in the target video frame can be obtained, that is, the boundary position of the macroblock boundary in the target video frame can be obtained.

[0147] Optionally, the image processing device obtains the boundary map E by determining the boundary position in the target video frame, for example, Figure 4 The image shown is a boundary map E, which is determined by Figure 2 The boundary position in the target video frame is obtained. Figure 4 In the boundary diagram E shown, the black line indicates the location where the block effect may occur in the target video frame. Specifically, the black line is the boundary of the macroblock. It should be understood that the boundary between two adjacent macroblocks does not overlap, so the black line includes two rows of pixels, or two columns of pixels, that is, the black line includes the boundary between two adjacent macroblocks.

[0148] Optionally, in the boundary map E, for any pixel p, E(p)=1 indicates that the pixel p is in the area covered by the black line, and E(p)=0 indicates that the pixel p is in the area not covered by the black line. Specifically, the pixel value in the boundary map E can be expressed by the following formula:

[0149]

[0150] The image processing device then calculates the blocking effect area of ​​the target video frame according to the boundary position in the boundary map E, thereby determining the blocking effect area from the target video frame (equivalent to step 103 described above), and obtaining the blocking effect map B, wherein the blocking effect map B includes the position information of the blocking effect area in the target video frame.

[0151] Optionally, when the pixel value in the boundary map E is represented by formula (2), and the image processing apparatus determines the blocking area in the target video frame by executing steps 201, 202, 203, 301, 302, and 303, the image processing apparatus may determine whether the pixel to be confirmed in the target video frame is a blocking pixel by the following formula:

[0152]

[0153] Among them, sum1 represents the target number, optional, Where N×N represents the preset size, E(q i )=1 indicates a boundary pixel, E(q i )=0 indicates non-boundary pixels. i represents the gradient of the i-th pixel in the pixel neighborhood constructed with the pixel to be confirmed as the center, represents the gradient sum of the boundary pixels in the neighborhood of the pixel, represents the gradient sum of non-boundary pixels in the neighborhood of the pixel, and α is the preset value in step 303.

[0154] Finally, after obtaining the blockiness map B, the image processing device performs a sharpening calculation on the target video frame according to the blockiness map B (the implementation process of the sharpening calculation can be referred to in step 701 described above), and obtains an enhanced video frame (i.e., Figure 3 The enhanced video frame A_pro in .

[0155] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0156] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among which, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0157] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0158] See also Figure 5 , Figure 5 The present invention provides a schematic diagram of the structure of an image processing device 1 provided in an embodiment of the present invention. The image processing device 1 includes an acquisition unit 11 and a determination unit 12. Optionally, the image processing device 1 also includes a processing unit 13. Specifically:

[0159] An acquisition unit 11 is used to acquire a target video frame and encoding information of the target video frame, wherein the encoding information includes a macroblock position of a macroblock in the target video frame;

[0160] A determination unit 12, configured to determine a boundary position of a macroblock boundary in the target video frame based on the macroblock position;

[0161] The determining unit 12 is further configured to determine a blocking effect region from the target video frame based on the boundary position, wherein the blocking effect region is a region where blocking effects occur.

[0162] In combination with any implementation manner of the present application, the determining unit 12 is used to:

[0163] Determining boundary pixels located on the boundary of the macroblock based on the boundary position;

[0164] Constructing a pixel neighborhood with a preset size with the pixel to be confirmed in the target video frame as the center;

[0165] determining a target number of said boundary pixels within said pixel neighborhood;

[0166] When the target number is greater than or equal to a first threshold, determining that the pixel to be confirmed is a blocking pixel;

[0167] The blocking effect area is determined according to the blocking effect pixels in the target video frame.

[0168] In combination with any implementation manner of the present application, the determining unit 12 is further configured to:

[0169] Determine the average gradient of the boundary pixels in the pixel neighborhood to obtain a first gradient;

[0170] Determine the average gradient of pixels in the pixel neighborhood except the boundary pixels to obtain a second gradient;

[0171] When the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value, it is determined that the pixel to be confirmed is the blocking pixel, and the preset value is greater than 1.

[0172] In combination with any implementation manner of the present application, the determining unit 12 is used to:

[0173] Determine the gradient sum of the boundary pixels in the pixel neighborhood by using a Sobel operator;

[0174] The quotient of the gradient sum of the boundary pixels and the target number is determined to obtain the first gradient.

[0175] In combination with any implementation manner of the present application, the determining unit 12 is used to:

[0176] Determine pixels in the target video frame whose distance to the boundary position is less than or equal to a second threshold as blocking pixels;

[0177] The blocking effect area is determined according to the blocking effect pixels in the target video frame.

[0178] In combination with any implementation manner of the present application, the determining unit 12 is used to:

[0179] determining a flat area from the target video frame;

[0180] The pixels in the flat area whose distance to the boundary position is less than or equal to a second threshold are determined as the blocking pixels.

[0181] In combination with any embodiment of the present application, the device further includes:

[0182] The processing unit 13 is configured to perform smoothing on the blocking area in the target video frame to obtain an enhanced video frame.

[0183] In combination with any embodiment of the present application, the processing unit 13 is used to:

[0184] The area other than the block effect area in the target video frame is sharpened, and the block effect area in the target video frame is smoothed to obtain the enhanced video frame.

[0185] In the embodiment of the present application, after acquiring the target video frame and the coding information of the target video frame, the image processing device uses the macroblock position in the coding information to determine the boundary position of the macroblock boundary in the target video frame. Because the blocking effect is usually easy to appear at the boundary of the macroblock, and the closer the area to the boundary of the macroblock is, the higher the probability of the blocking effect, the image processing device can determine the blocking effect area from the target video frame based on the boundary position, thereby determining the blocking effect area in the target video frame based on the coding information of the target video frame.

[0186] Since the encoding information of the target video frame is the information carried by the target video frame, determining the blocking effect area in the target video frame based on the encoding information of the target video frame can improve the efficiency of determining the blocking effect area in the target video frame and reduce the data processing amount of determining the blocking effect area in the target video frame.

[0187] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0188] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0189] The processor 21 may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Optionally, the processor 21 may be a processor group consisting of multiple CPUs, and the multiple processors are coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0190] The memory 22 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0191] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0192] It can be understood that in the embodiment of the present application, the memory 22 can be used not only to store relevant instructions, but also to store relevant data. The embodiment of the present application does not limit the specific data stored in the memory.

[0193] Understandably, Figure 6 Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0194] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0195] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0196] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0197] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0199] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0200] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. An image processing method, It is characterized in that The method comprises: Acquire a target video frame and encoding information of the target video frame, wherein the encoding information includes a macroblock position of a macroblock in the target video frame; Determine the boundary position of the boundary of the macroblock in the target video frame based on the macroblock position; A blocking effect area is determined from the target video frame based on the boundary position, where the blocking effect area is an area where blocking effect occurs; the determining of the blocking effect area from the target video frame based on the boundary position includes: determining boundary pixels located on the boundary of the macroblock based on the boundary position; constructing a pixel neighborhood with a preset size centered on the pixel to be confirmed in the target video frame; determining a target number of the boundary pixels in the pixel neighborhood; determining the pixel to be confirmed as a blocking effect pixel when the target number is greater than or equal to a first threshold; and determining the blocking effect area according to the blocking effect pixels in the target video frame.

2. The method according to claim 1, It is characterized in that Before determining that the pixel to be confirmed is a blocking pixel when the target number is greater than or equal to the first threshold, the method further includes: Determine the average gradient of the boundary pixels in the pixel neighborhood to obtain a first gradient; Determine the average gradient of pixels in the pixel neighborhood except the boundary pixels to obtain a second gradient; The step of determining that the pixel to be confirmed is a blocking pixel when the target number is greater than or equal to a first threshold comprises: When the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value, it is determined that the pixel to be confirmed is the blocking pixel, and the preset value is greater than 1.

3. The method according to claim 2, It is characterized in that The determining the average gradient of the boundary pixels in the pixel neighborhood to obtain a first gradient includes: Determine the gradient sum of the boundary pixels in the pixel neighborhood by using a Sobel operator; The quotient of the gradient sum of the boundary pixels and the target number is determined to obtain the first gradient.

4. The method according to any one of claims 1 to 3, It is characterized in that The determining the blocking effect area according to the blocking effect pixels in the target video frame includes: A pixel region including the blocking effect pixels in the target video frame is used as the blocking effect region.

5. The method according to claim 4, It is characterized in that The step of taking a pixel region including the blocking effect pixels in the target video frame as the blocking effect region comprises: A pixel region composed of the blocking effect pixels in the target video frame is used as the blocking effect region.

6. The method according to claim 2, It is characterized in that The step of determining an average gradient of pixels in the pixel neighborhood except the boundary pixels to obtain a second gradient comprises: Determine the gradient sum of non-boundary pixels in the pixel neighborhood by using a Sobel operator; determining the number of said non-boundary pixels within said pixel neighborhood; The quotient of the gradient sum of the non-boundary pixels and the number of the non-boundary pixels is determined to obtain the second gradient.

7. The method according to claim 2 or 3, It is characterized in that The step of determining that the pixel to be confirmed is the blocking pixel when the target number is greater than or equal to a first threshold and the first gradient is greater than a product of the second gradient and a preset value includes: In a case where the first gradient is greater than a product of the second gradient and the preset value, determining whether the target quantity is greater than or equal to the first threshold; When the target number is greater than or equal to the first threshold, the pixel to be confirmed is determined to be the blocking pixel.

8. An image processing device, It is characterized in that The device comprises: An acquisition unit, configured to acquire a target video frame and encoding information of the target video frame, wherein the encoding information includes a macroblock position of a macroblock in the target video frame; A determination unit, configured to determine a boundary position of a macroblock boundary in the target video frame based on the macroblock position; The determining unit is further configured to determine a blocking effect region from the target video frame based on the boundary position, wherein the blocking effect region is a region where blocking effects occur; the determining the blocking effect region from the target video frame based on the boundary position comprises: determining boundary pixels located on the boundary of the macroblock based on the boundary position; constructing a pixel neighborhood with a preset size centered on the pixel to be confirmed in the target video frame; determining a target number of the boundary pixels in the pixel neighborhood; determining the pixel to be confirmed to be a blocking effect pixel when the target number is greater than or equal to a first threshold; and determining the blocking effect region according to the blocking effect pixels in the target video frame.

9. An electronic device, It is characterized in that include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 7.