Image processing method, apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to an image processing method, device, equipment and medium, wherein the method comprises: acquiring a plurality of groups of image pairs with different grid resolutions; according to the order of the grid resolutions from small to large, sequentially calculating the grid vertex positions of the corresponding second images according to the corresponding second images and the associated reference images of each group of image pairs in the plurality of groups of image pairs, and sequentially passing the grid vertex positions of the previous grid resolution to the next layer until the target grid deformation image of the image pair corresponding to the maximum grid resolution is acquired, and calculating the grid flow according to the target grid deformation image and the first image, so as to align the original reference image and the first image according to the grid flow. Thus, the grid vertex positions of the previous pair of images are passed from coarse to fine according to the order of the grid resolutions from small to large to control the grid shape deformation, and the grid flow is determined according to the deformed grid, thereby improving the acquisition efficiency and robustness of the grid flow and improving the image alignment effect.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and medium. Background Technology

[0002] Mesh flow is an image alignment technique that involves dividing the image to be aligned into a checkerboard pattern, then deforming the image by moving the grid vertices, and finally aligning the deformed target image to a reference image. Mesh flow is generally used in scenarios requiring motion estimation between video sequences, such as video denoising, image stabilization, and, for example, aligning image elements between images by placing elements from one image into the corresponding image regions of another, thus adding special effects to the other image.

[0003] In related technologies, feature point matching is performed based on grid flow technology. That is, after determining the matching feature points between the image to be aligned and the reference image, the displacement of the grid vertices is calculated based on the displacement of the feature points. Then, the grid vertices of the image to be aligned are moved by the displacement to achieve image alignment.

[0004] However, the above-mentioned image alignment method based on feature points requires the detection and matching of feature points, which results in a long processing time and affects the efficiency of image alignment. Furthermore, feature points in image regions with weak local textures are difficult to measure, which leads to large errors in the calculation of mesh vertex displacement, and thus affects the quality of image alignment. Summary of the Invention

[0005] To address or at least partially address the aforementioned technical problems, this disclosure provides an image processing method. The method includes: acquiring multiple sets of image pairs with different grid resolutions, wherein each set of image pairs includes a second image and an associated reference image, wherein the second image is obtained by meshing a first image at a corresponding grid resolution, and the associated reference image is obtained by scaling the original reference image to be aligned with the first image according to the image size of the second image; updating the grid vertex positions of the corresponding second image based on the corresponding second image and the associated reference image when the current image pair has the smallest grid resolution, according to the grid resolution in ascending order; updating the grid vertex positions of the corresponding second image in the current image pair based on the grid vertex positions of the previous image pair, and updating the grid vertex positions of the corresponding second image based on the corresponding second image and the associated reference image when the current image pair has the largest grid resolution; acquiring a target grid deformation image based on the grid vertex positions of the corresponding second image in the current image pair, and calculating a grid flow based on the target grid deformation image and the first image, so as to align the original reference image and the first image according to the grid flow.

[0006] This disclosure also provides an image processing apparatus, comprising: an acquisition module for acquiring multiple sets of image pairs with different grid resolutions, wherein each set of image pairs includes a second image and an associated reference image, wherein the second image is acquired by meshing a first image at a corresponding grid resolution, and the associated reference image is acquired by scaling the original reference image to be aligned with the first image according to the image size of the second image; an update module for updating the grid vertex positions of the corresponding second image according to the grid resolution in ascending order, when the current image pair has the minimum grid resolution, based on the corresponding second image and the associated reference image; the update module is further configured to update the corresponding grid vertex positions of the second image of the current image pair according to the grid vertex positions of the previous image pair when the current image pair does not have the minimum grid resolution, and to update the corresponding grid vertex positions of the second image of the corresponding second image according to the corresponding second image and the associated reference image; and an alignment calculation module for acquiring a target grid deformation image according to the grid vertex positions of the corresponding second image of the current image pair when the current image pair has the maximum grid resolution, and calculating a grid flow according to the target grid deformation image and the first image, so as to align the original reference image and the first image according to the grid flow.

[0007] This disclosure also provides an electronic device, which includes: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the image processing method provided in this disclosure.

[0008] This disclosure also provides a computer-readable storage medium storing a computer program for performing the image processing method provided in this disclosure.

[0009] Compared with the prior art, the technical solution provided in this disclosure has the following advantages: The image processing solution provided in this disclosure acquires multiple sets of image pairs with different grid resolutions. Based on the grid resolution from smallest to largest, it sequentially calculates the grid vertex positions of the corresponding second image in each set of image pairs according to the second image and the associated reference image. The grid vertex positions of the previous grid resolution are then sequentially passed to the next layer until the target grid deformed image corresponding to the image pair with the largest grid resolution is obtained. A grid flow is then calculated based on the target grid deformed image and the first image to facilitate alignment of the original reference image and the first image according to the grid flow. Therefore, by controlling the grid shape deformation by passing the grid vertex positions of the previous image pair from coarse to fine according to the grid resolution from smallest to largest, and determining the grid flow based on the deformed grid, the efficiency and robustness of grid flow acquisition are improved, thus enhancing the image alignment effect. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of the present disclosure;

[0012] Figure 2(a) is a scene diagram of an image pair provided in an embodiment of this disclosure;

[0013] Figure 2(b) is a scene diagram of another image pair provided in an embodiment of this disclosure;

[0014] Figure 2(c) is a scene diagram of another image pair provided in an embodiment of this disclosure;

[0015] Figure 3(a) is a schematic diagram of an image pair generation scenario provided by an embodiment of this disclosure;

[0016] Figure 3(b) is a schematic diagram of another image pair generation scenario provided by an embodiment of this disclosure;

[0017] Figure 4(a) is a schematic diagram of another image alignment scenario provided by an embodiment of this disclosure;

[0018] Figure 4(b) is a schematic diagram of another image alignment scenario provided by an embodiment of this disclosure;

[0019] Figure 5 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0020] Figure 6 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0021] Figure 7 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0022] Figure 8 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0023] Figure 9 This is a schematic diagram of a mesh deformation scene provided in an embodiment of the present disclosure;

[0024] Figure 10 This is a schematic diagram of a mesh similarity transformation scene provided in an embodiment of the present disclosure;

[0025] Figure 11 This is a schematic diagram of another image alignment scenario provided by an embodiment of the present disclosure;

[0026] Figure 12 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0027] Figure 13(a) is a schematic diagram of another image alignment process provided in an embodiment of this disclosure;

[0028] Figure 13(b) is a schematic diagram of the grid flow of a pixel provided in an embodiment of this disclosure;

[0029] Figure 14 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure;

[0030] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0031] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0032] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0033] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0034] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0035] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0036] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0037] To address the aforementioned issues, this disclosure provides an image processing method. In this method, to handle large displacements between the image to be aligned and a reference image, multiple grid resolutions within a multi-layer pyramid structure are constructed. Different numbers of grid divisions are set for each layer of the image based on these resolutions. The optimization results corresponding to other grid resolutions are passed from coarse to fine, and finally, a grid flow is determined. This ensures the efficiency and robustness of the grid flow determination, thereby improving the image alignment effect.

[0038] The method will be described below with reference to specific embodiments.

[0039] Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of the present disclosure. The method can be executed by an image processing device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes:

[0040] Step 101: Obtain multiple sets of image pairs with different grid resolutions.

[0041] In this embodiment, each image pair includes a second image and an associated reference image. The grid resolution of each image pair is different. The second image is obtained by dividing the first image into a grid at the corresponding grid resolution. The associated reference image is obtained by scaling the original reference image to be aligned with the first image based on the image size of the second image. In this embodiment, the image processing aims to align the first image and the original reference image. The original reference image and the first image can be adjacent video frames in the video stream, or non-adjacent video frames in the same video stream. In the foreground completion scenario, the original reference image can be understood as the image to be completed for the foreground, and the first image can be understood as the image to complete the foreground, etc., without limitation.

[0042] In one embodiment of this disclosure, a first image can be divided into multiple second images with different grid resolutions according to a preset unit grid size and multiple grid resolutions.

[0043] In this embodiment, the preset unit grid size is usually smaller than the first image size, equivalent to the size of one grid cell in the grid image after grid division. The preset unit grid size can be calibrated based on experimental data.

[0044] To handle cases where pixel displacement is large, such as large displacement between the original reference image and the first image, multiple grid resolutions can be preset, and the first image can be divided into grids to generate multiple second images with different grid resolutions.

[0045] As shown in Figure 2(a), multiple grid resolutions can be arranged in a pyramid structure, meaning that after arranging the corresponding second images according to size, the ratio of grid resolutions in adjacent second images is the same. In some possible embodiments, as shown in Figure 2(b), after arranging the corresponding second images according to ascending order of grid resolutions, the ratio between adjacent second images of grid resolutions may also be different.

[0046] In this embodiment, the first image is first divided into grids to generate multiple second images with different grid resolutions. The number of unit grids in each second image is consistent with the corresponding grid resolution. For example, as shown in Figure 3(a), when multiple grid resolutions correspond to three grid resolutions, namely “1×1”, “2×2” and “4×4”, the corresponding second images contain chessboard structures with 1 unit grid, 4 unit grids and 16 unit grids, respectively.

[0047] Then, the original reference image is scaled according to the image size of the second image corresponding to each grid resolution to generate a size-matched associated reference image.

[0048] In this embodiment, in order to achieve image alignment at various coarse and fine granularities and to take into account various displacement degrees for subsequent alignment processing, the original reference image is scaled according to the image size of the second image corresponding to each grid resolution to generate a size-matched associated reference image. It can also be understood that the second image corresponding to each grid resolution has an associated reference image of the same size. The associated reference image is only consistent with the size of the corresponding second image, and the associated reference image does not need to be gridded.

[0049] Continuing with the scenario shown in Figure 3(a), as shown in Figure 3(b), the three second images correspond to three associated reference images of the same size. Therefore, for each second image corresponding to a grid resolution, an associated reference image of the same size is set to be matched with it. The second image and the associated reference image at the same grid resolution form an image pair, thereby improving the computational efficiency of image alignment.

[0050] Step 102: According to the order of grid resolution from smallest to largest, when the current image pair has the smallest grid resolution, update the grid vertex position of the corresponding second image based on the corresponding second image and the associated reference image.

[0051] In this embodiment, it should be understood that the alignment of the second image and the associated reference image can be achieved by displacing the grid vertices to deform the grid in the second image. The grid deformation causes the displacement of the pixels in the second image, so that the second image after grid deformation and the associated reference image are aligned at the corresponding grid resolution.

[0052] Taking a second image with a grid resolution of 1×1 as an example, the second image contains a grid A1, whose corresponding associated reference image is B. Then, for A1, the coordinate vectors of its four grid vertices are respectively... , , and Each pixel within grid A1 can be represented by the coordinate vector of the vertex of its grid. Taking pixel q as an example, the coordinate vector of the pixel's position can be expressed as follows: (1) where, in the following formula... The weights for each grid vertex can be preset:

[0053] Formula (1)

[0054] Furthermore, when aligning the second image with the corresponding associated reference image, the coordinate vectors of the determined grid vertices are as follows: , , and After controlling the deformation of mesh A1 into A2 by the vertex of the mesh, continue to use the pixels of the aforementioned pixel q in A2. For example, this pixel The coordinate vector of the position can be expressed as follows (2):

[0055] Formula (2)

[0056] Therefore, it can be understood that in this embodiment, by aligning the second image and the corresponding associated reference image, the positions of the grid vertices of the second image are updated. The updated grid vertex positions are the deformed grid vertices. Therefore, the deformation of the grid in the second image can obviously be controlled according to the deformation of the grid vertices. The deformed grid drives the movement of the pixels inside the grid, which actually realizes the movement and alignment of the pixels inside the grid. Thus, in this embodiment, there is no need to detect and calculate the displacement of the feature points between the second image to be aligned and the associated reference image, which greatly improves the alignment efficiency.

[0057] In this embodiment, since the calculation is performed in order of increasing resolution and in order of coarse to fine displacement granularity, the purpose of the calculation is to obtain the result of the mesh deformation of the second image with the maximum mesh resolution. Therefore, in this embodiment, if the current image pair has the minimum mesh resolution, the mesh vertex position of the corresponding second image can be updated simply based on the corresponding second image and the associated reference image, without needing to control the actual deformation of the second image.

[0058] The updated grid vertex position can be understood as the position of the network vertex of the associated reference image after the second image is aligned with the displacement granularity at the corresponding grid resolution. The grid vertex position can be understood as the coordinate vector of the grid vertex, etc.

[0059] Step 103: When the current image pair is not the minimum grid resolution, update the grid vertex position of the second image of the current image pair according to the grid vertex position of the previous image pair, and update the grid vertex position of the corresponding second image according to the corresponding second image and the associated reference image.

[0060] In this embodiment, the grid vertex positions of the second image in the current image pair are updated according to the grid vertex positions of the previous image pair, in order of increasing grid resolution. That is, the aligned grid vertex positions at each grid resolution are passed sequentially. The grid vertex positions correspond to the deformation of the grid shape at each grid resolution. Therefore, in this embodiment, the deformed grid shape is passed to the last second image in order of increasing grid resolution.

[0061] In this embodiment, if the current image pair is not at the minimum grid resolution, the grid vertex position of the second image corresponding to the current image pair is updated according to the grid vertex position of the previous image pair. The grid vertex position of the previous image pair is the position of the already aligned coarser-grained grid vertex after deformation. Therefore, by combining the grid vertex positions of the previous image pair, the deformation of the previously aligned coarser-grained grid shape is transferred to the second image at the current grid resolution, further improving the image alignment efficiency.

[0062] In actual execution, in order to facilitate the transmission of the grid vertex positions of the previous image pair, it is also necessary to determine the grid vertex positions of the previous image pair corresponding to the grid vertices in the current image pair.

[0063] In this embodiment, to determine the position of the corresponding grid vertex in the second image of the current image pair, the grid resolution ratio between the current image pair and the previous image pair can be calculated. This resolution ratio is used to indicate the coordinate ratio of each grid vertex in the adjacent image pair. Then, after determining the corresponding grid vertex of each grid vertex in the second image of the current image pair in the previous image pair, since the resolution ratio is used to indicate the coordinate ratio of each grid vertex in the adjacent image pair, the grid vertex position of each grid vertex in the second image of the current image pair is updated according to the product of the updated grid vertex position of the corresponding grid vertex and the grid resolution ratio.

[0064] However, in practical applications, determining the position of each grid vertex after the grid vertex update may require obtaining the weighted values ​​of the three other grid vertices belonging to the same unit grid. In order to ensure the stability of grid deformation, the weighted values ​​of the grid vertices are the same as the weighted values ​​after the grid vertex positions are not updated. Therefore, in one embodiment of this disclosure, the original grid vertex positions of the grid vertices of the second image in the previous image pair before the grid vertex positions are updated can also be determined according to the grid resolution ratio. The weighted values ​​between the grid vertices are then calculated based on these vertex positions to facilitate the subsequent determination of the position of each grid vertex after the update.

[0065] To facilitate understanding, the following example illustrates the process of updating the grid vertex position of the second image in the current image pair based on the grid vertex position of the previous image pair, using a ratio of 2 in both the horizontal and vertical coordinates.

[0066] In this embodiment, when multiple grid resolutions of multiple image pairs correspond to L (L is 3 in this embodiment) grid resolutions, namely "1×1", "2×2" and "4×4", the second images and associated reference images between multiple image pairs can be arranged in order of increasing grid resolution to construct an L-layer image pyramid structure.

[0067] If the current image corresponds to layer L of the pyramid structure, where L is greater than 1, then the mesh shape of layer L-1 is propagated to the current layer. Let... This represents the grid corresponding to the second image of layer L. This represents the original grid corresponding to the second image in layer L-1. This represents the deformed mesh corresponding to the second image of layer L-1 (this deformed mesh can refer to the mesh after updating the mesh vertex positions based on the second image of layer L-1 and the associated reference image).

[0068] Traversal Each grid vertex in Since the resolution of the lower grid is twice that of the upper grid in adjacent grid resolutions, the grid vertices can be obtained according to formula (3). exist The coordinates of the corresponding grid vertex :

[0069] Formula (3)

[0070] Furthermore, based on bilinear interpolation algorithms, etc., it is possible to obtain exist Interpolation information in the middle to obtain The four grid vertices of the corresponding cell and the weight of each grid vertex , It can be represented as a weighted linear combination of the four vertices, as shown in formula (4);

[0071] Formula (4)

[0072] Furthermore, after obtaining the weight of each grid vertex, the weight is calculated according to formula (5). Corresponding mesh vertices ,in, for exist The corresponding mesh vertices:

[0073] Formula (5)

[0074] Furthermore, the positions of the original mesh vertices corresponding to the second image of the current image pair are determined according to formula (6). Update the mesh vertex positions based on the mesh vertex positions of the previous image pair. :

[0075] Formula (6)

[0076] This cycle is repeated to achieve the transfer of deformable mesh shape from coarse to fine.

[0077] For example, as shown in Figure 2(c), when multiple grid resolutions correspond to L (L is 3 in this embodiment) grid resolutions, namely "1×1", "2×2" and "4×4", a pyramid structure layer of the second image corresponding to L grid resolutions can be constructed in ascending order. First, the grid vertex positions of the second image in the first layer are determined based on the first layer second image and the associated reference image. Then, the initial grid vertex positions of the second image in the second layer are updated based on the grid vertex positions of the two images. The grid vertex positions of the second image in the second layer are determined for the updated second image and the associated reference image in the second layer. At this time, the grid vertex positions are the result of the combined effect of the alignment of the first layer grid vertex and the alignment of the second layer grid vertex.

[0078] Similarly, the initial mesh vertex positions of the second image in the third layer are updated based on the mesh vertex positions of the second image in the second layer. The mesh vertex positions of the second image in the third layer are then determined using the updated second image and the associated reference image in the third layer. At this point, the mesh vertex positions of the second image in the third layer are the result of the combined effects of mesh vertex alignment in the second and third layers. This achieves the transfer of mesh deformation shapes from the first layer to the second and third layers.

[0079] Step 104: When the current image pair has the maximum grid resolution, obtain the target grid deformation image based on the grid vertex position of the second image corresponding to the current image pair, and calculate the grid flow based on the target grid deformation image and the first image, so as to align the original reference image and the first image according to the grid flow.

[0080] As mentioned above, in this embodiment, the sensitivity of the determined grid vertex positions to the degree of displacement varies depending on the grid resolution. Therefore, in order to integrate displacements of multiple granularities for image alignment, in this embodiment, the second image corresponding to the maximum grid resolution is adjusted according to the alignment displacement information of the grid vertices corresponding to each grid resolution to obtain the target grid deformation image. That is, the displacement of the grid vertices under all displacement degrees is ultimately reflected in the grid shape at the maximum grid resolution in this embodiment. In other words, based on the grid vertex positions of the current image and the corresponding second image, the displacement of each pixel within the grid can be known, and thus, the target grid deformation image can be obtained.

[0081] In this embodiment, since the grid flow is good at tracking static backgrounds, the grid flow mainly reflects the motion field of the background pixels between the original reference image and the first image. In this embodiment, the grid flow can be calculated based on the target grid deformation image and the first image. That is, the motion field corresponding to the deformation of the first image to the target grid deformation image is the grid flow. Then, the original reference image and the first image are aligned according to the grid flow. The alignment of the original reference image and the first image can be: controlling the deformation of the first image according to the grid flow, or denoising the first image according to the grid flow, etc. Of course, in some scenarios, a better synthesis effect can also be achieved by compositing the denoised first image and the original reference image to avoid synthesizing "ghosting".

[0082] In practical applications, aligning the original reference image and the first image based on the mesh flow can improve video quality in scenarios such as video denoising, and enhance the visual appeal of images in scenarios such as adding special effects. When adding special effects, the mesh flow of the image region containing the element to be added to the special effect in the original reference image is determined. Based on the mesh flow, the texture pixels of the corresponding element in the first image are aligned with the corresponding image region in the original reference image, achieving a natural presentation of visual effects.

[0083] It should be understood that special effects can be applied in any scenario, such as aligning advertising elements to buildings, or composite elements of other objects onto clothing. Examples are provided below:

[0084] For example, as shown in Figure 4(a), mesh flow alignment can be used to blend elements from the fish tank onto the user's clothes according to the angle of the clothes. In the first image, the element to be aligned in the original reference image is the "fish" in the dashed line, and the element to be aligned in the original reference image is a blank area marked by the dashed line in the clothes. By aligning based on mesh flow, the effect of attaching the "fish" in the fish tank to the clothes can be achieved. The "fish" on the clothes fits the shape of the corresponding area on the clothes, achieving a natural effect addition.

[0085] For example, as shown in Figure 4(b), grid flow alignment can be used to integrate elements from a beverage bottle into a book, making the aligned image appear as if the corresponding elements were originally written into the book. In the first image, the element to be aligned in the original reference image is the "star" in the dashed line, and the element to be aligned in the original reference image is a blank area marked by the dashed line in the book. By aligning based on grid flow, the effect of attaching the "star" on the beverage bottle to the book can be achieved. The "star" on the book fits the shape of the corresponding area on the book, achieving a natural effect addition.

[0086] It should be noted that the method for calculating the mesh flow based on the target mesh deformation image and the first image differs in different application scenarios:

[0087] In some possible embodiments, a deep learning model is learned in advance using deep learning technology, and the corresponding mesh flow is output based on the input target mesh deformation image and the first image using the deep learning model.

[0088] In some possible embodiments, such as Figure 5 As shown, the mesh flow is calculated based on the target mesh deformation image and the first image, including:

[0089] Step 501: Based on the positions of the grid vertices in the target mesh deformation image, obtain the first pixel position of each pixel in the target mesh deformation image.

[0090] As mentioned above, the pixel position of each pixel can be represented by the vertex coordinates of the grid it belongs to. Therefore, by determining the grid vertex positions in the target grid deformation image, the first pixel position of each pixel in the target grid deformation image can be obtained.

[0091] Step 502: Obtain the second pixel position of each pixel based on the initial grid vertex position of the second image corresponding to the maximum grid resolution.

[0092] The original second image here can be understood as the second image corresponding to the grid resolution when the grid was initially divided.

[0093] In this embodiment, the second pixel position of each pixel in the original second image is obtained based on the initial grid vertex position of the second image corresponding to the maximum grid resolution. Similarly, the second pixel position of each pixel can be determined based on the vertex coordinates of the grid vertex where the pixel is located in the original second image.

[0094] Step 503: Calculate the displacement of the second pixel position and the corresponding first pixel position of each pixel to obtain the mesh flow.

[0095] In fact, the grid flow can be regarded as the displacement motion field of pixels during the image alignment process. Therefore, in this embodiment, the grid flow is obtained by calculating the displacement difference between the second pixel position and the corresponding first pixel position of each pixel.

[0096] In summary, the image processing method of this disclosure acquires multiple sets of image pairs with different grid resolutions. Based on the grid resolution from smallest to largest, it sequentially calculates the grid vertex positions of the corresponding second image in each set of image pairs using the second image and associated reference image. The grid vertex positions of the previous grid resolution are then passed to the next layer until the target grid deformed image corresponding to the image pair with the largest grid resolution is obtained. A grid flow is then calculated based on the target grid deformed image and the first image to facilitate alignment of the original reference image and the first image. Therefore, by controlling the grid shape deformation by passing the grid vertex positions of the previous image pair from coarse to fine according to the grid resolution from smallest to largest, and determining the grid flow based on the deformed grid, the efficiency and robustness of grid flow acquisition are improved, thus enhancing the image alignment effect.

[0097] In practical applications, the mesh vertex positions of the second image and the associated reference image in each image pair are calculated, thereby achieving image alignment at the corresponding displacement granularity. The method for determining the mesh vertex positions of the corresponding second image based on the corresponding second image and the associated reference image varies in different application scenarios, as shown in the following examples:

[0098] In one embodiment of this disclosure, a deep learning model is pre-trained based on a large amount of sample data. The input of the deep learning model is two images to be aligned, and the output is the grid vertex position of the target image to be aligned. Therefore, in this embodiment, the corresponding second image and the associated reference image are input into the pre-trained deep learning model to obtain the grid vertex position of the corresponding second image.

[0099] In another embodiment of this disclosure, in order to further improve the image alignment effect, the grid vertex positions of the second image are iteratively calculated for each pair of images.

[0100] like Figure 6As shown, the mesh vertex positions of the corresponding second image are determined based on the corresponding second image and the associated reference image, including:

[0101] Step 601: Based on the grid vertex positions of the corresponding second image, iteratively calculate the residual values ​​between the corresponding second image and the associated reference image.

[0102] Here, the residual value can be understood as the vertex alignment error between the second image and the associated reference image. The grid vertex position of the corresponding second image is updated based on this alignment error to achieve the gradual alignment between the second image and the associated reference image.

[0103] In different application scenarios, the residual value includes different contents. In some possible embodiments, the residual value includes photometric error value, that is, it includes the brightness error of the pixels of the second image and the associated reference image, thereby determining the vertex alignment error of the second image and the associated reference image based on the brightness dimension.

[0104] In some other possible embodiments, the residual value includes a deformation error value that ensures that the squares within the mesh conform to a similar transformation as much as possible when deformed, thereby determining the vertex alignment error between the second image and the associated reference image based on the shape dimension.

[0105] In some other possible embodiments, the vertex alignment error between the second image and the associated reference image can be determined by combining the aforementioned photometric error value and deformation error value.

[0106] Step 602: During each iteration, determine whether the residual value of the current iteration meets the preset iteration stop condition.

[0107] In this embodiment, to avoid slow image alignment efficiency due to too many iterations, an iteration stop condition is set. Once the iteration stop condition is met, the iteration calculation of the current image pair is stopped, and the alignment calculation of the next set of image pairs begins.

[0108] The preset iteration stopping condition varies in different application scenarios:

[0109] In one embodiment of this disclosure, a preset number of iterations matching the grid resolution is set for each image pair. Iteration stops when the preset number of iterations is greater than or equal to this preset number. For example, when the grid resolution is 1×1, the preset number of iterations can be a value between 5 and 10; when the grid resolution is 2×2, the preset number of iterations can be a value between 3 and 5; and when the grid resolution is 4×24, the preset number of iterations can be a value between 5 and 10.

[0110] In another embodiment of this disclosure, the iteration stopping condition is determined based on the iteration performance. That is, in this embodiment, it is determined whether the residual value of the iteration calculation is greater than or equal to the residual value of the previous iteration calculation. If it is greater than or equal to the residual value of the previous iteration calculation, it means that the iteration performance has deteriorated. Therefore, it is determined that the iteration stopping condition is met and the iteration calculation is stopped.

[0111] In another embodiment of this disclosure, in order to balance iteration efficiency and iteration performance, the iteration stopping condition is determined by combining the above two embodiments.

[0112] In this embodiment, as Figure 7 As shown, determining whether the residual value calculated in the current iteration meets the preset iteration stopping condition includes:

[0113] Step 701: Determine whether the number of iterations calculated in the current iteration is greater than or equal to the preset number of iterations for the current image pair. The preset number of iterations is greater than 1.

[0114] Step 702: If the number of iterations is greater than or equal to the preset number of iterations, then the residual value calculated in the current iteration is determined to meet the preset iteration stopping condition.

[0115] In this embodiment, it is assumed that the current iteration calculation number is . The preset number of iterations is ,if Greater than or equal to If the preset iteration stopping condition is met, then it is considered that the iteration stopping condition is satisfied.

[0116] Step 703: If the number of iterations is not greater than or equal to the preset number of iterations, then obtain the historical residual value of the previous iteration for the current image pair.

[0117] Step 704: Determine whether the residual value calculated in the current iteration is greater than or equal to the historical residual value.

[0118] Step 705: If the residual value is greater than or equal to the historical residual value, then the residual value calculated in the current iteration is determined to meet the preset iteration stop condition.

[0119] In this embodiment, if the number of iterations is not greater than or equal to a preset number of iterations, the historical residual value of the previous iteration corresponding to the current image pair is obtained. If the residual value of the current iteration is greater than or equal to the historical residual value, it is determined that the performance of the current iteration has decreased, thereby determining that the residual value of the current iteration satisfies the preset iteration stopping condition.

[0120] For example, when the historical residual value is The residual value of the current image pair is Then when Greater than or equal to When the residual value of the current iteration is satisfied with the preset iteration stopping condition, it is determined that the residual value is satisfied with the current iteration.

[0121] Step 603: If the iteration stopping condition is not met, update the grid vertex position of the corresponding second image according to the residual value, and continue iterative calculation until the residual value of the current iteration calculation meets the preset iteration stopping condition.

[0122] In this embodiment, since the residual value is the alignment error, the grid vertex position of the corresponding second image is updated according to the residual value, and the iteration calculation continues until the residual value calculated in the current iteration meets the preset iteration stop condition.

[0123] Of course, in this embodiment, iterative calculation is not necessary. Instead, the residual value between the corresponding second image and the associated reference image can be calculated only once. In this case, the grid vertex position of the second image is updated based on the residual value, according to the original image of the second image.

[0124] To enable those skilled in the art to more clearly understand how to iteratively calculate the residual values ​​of the corresponding second image and the associated reference image based on the grid vertex positions of the corresponding second image, the following explanation will focus on the residual values ​​including the aforementioned photometric error values ​​and deformation error values.

[0125] In one embodiment of this disclosure, such as Figure 8 As shown, based on the grid vertex positions of the corresponding second image, the residual values ​​between the corresponding second image and the associated reference image are iteratively calculated, including:

[0126] Step 801: Calculate the photometric error value between the corresponding second image and the associated reference image based on the grid vertex position of the corresponding second image.

[0127] In this embodiment, the photometric error value can be determined based on the brightness difference between all pixels in the second image and their corresponding pixels in the associated reference image. However, when there are a large number of pixels, the computational load may be large, affecting the image alignment efficiency.

[0128] Therefore, in one embodiment of this disclosure, multiple sample pixels in the second image are determined, and the image alignment efficiency is improved by reducing the number of pixels used to calculate the photometric error value.

[0129] In this embodiment, firstly, multiple sample pixels of the corresponding second image are determined, and then, the grid where each sample pixel is located is determined. The pixel position of each sample pixel is determined according to the grid vertex position of the grid where it is located (as mentioned in the example above, the position of a pixel can be obtained by weighting the grid vertex position of its cell).

[0130] The sample pixels mentioned above can be randomly determined or selected based on brightness values. In this embodiment, in order to further reduce the number of pixels used to calculate photometric error values ​​without affecting the image alignment effect, a uniform random sampling technique is introduced to further reduce the number of sample pixels used to construct photometric error values.

[0131] In this embodiment, the pixel gradient value of each pixel in the second image at the current grid resolution is traversed one by one according to a preset step size. Pixels whose pixel gradient value is greater than or equal to the preset gradient threshold are determined as candidate sample pixels. That is, if the gradient of the second image at the current grid resolution is divided into horizontal gradients... and gradient in the vertical direction Then, if the gradient value of pixel q in the horizontal direction gradient values ​​in the numerical direction The sum of these values ​​is greater than or equal to the preset gradient threshold. If the gradient threshold is set, the corresponding pixel is determined as a candidate sample pixel. In some possible embodiments, the preset gradient threshold can be between 0.05 and 2.

[0132] In one embodiment of this disclosure, in order to further improve the sample collection efficiency, after determining the number of candidate samples, the total number of candidate sample pixels can also be determined. When the total number is greater than or equal to the total number, the determination of candidate sample pixels is stopped, and all candidate sample pixels are used as sample pixels.

[0133] Alternatively, in some other possible embodiments, if the total number of candidate sample pixels is greater than a preset threshold (for example, the total number can be between 800 and 1000), the candidate sample pixels of the preset threshold are randomly determined as the sample pixels. For example, after randomly arranging all the candidate sample pixels, the candidate sample pixels belonging to the preset threshold in the random arrangement result are determined as multiple sample pixels. Thus, this random sampling strategy can significantly reduce the sample pixel size while ensuring the uniform distribution of samples on the image, improving computational efficiency while maintaining tracking performance.

[0134] Furthermore, after determining the sample pixels, the pixel position of each sample pixel is determined based on the grid where each sample pixel is located and the position of the grid vertex. The photometric error value is then determined based on the preset photometric error calculation algorithm and the pixel position.

[0135] It should be noted that the preset photometric error calculation algorithm in this embodiment can be any existing method for calculating photometric error, and no limitation is imposed here.

[0136] For example, the preset photometric error calculation algorithm is formula (7), where, in formula (7), This is the photometric error value. and These are the brightness values ​​of sample pixel q in the second image and the associated reference image, respectively. Let be the gradient of the sample pixel q in the second image, and let the four grid vertices of the cell containing q be... (k=1,2,3,4) The weights for each grid vertex can be preset.

[0137] Formula (7)

[0138] In this embodiment, as Figure 9 As shown, the mesh deformation process before and after updating the mesh vertex positions (where solid quadrilaterals represent the cell mesh before the update and dashed quadrilaterals represent the cell mesh after the update) is illustrated. Mesh deformation is achieved through vertex displacement, and the interpolation coefficients q before and after deformation are shown. Keeping the position unchanged, the mesh deformation moves the position of q to a more suitable location. This leads to a reduction in photometric error.

[0139] Step 802: Calculate the regular expression value of the corresponding second image based on the grid vertex position of the corresponding second image.

[0140] Among them, the regular expression value can be regarded as the deformation error value to ensure that the squares in the grid conform to the similarity transformation as much as possible when they are deformed. Based on the regular expression value, the image filled in the foreground area of ​​the aligned second image can look more natural and more in line with the perspective transformation.

[0141] In this embodiment, if the grid vertex position of the corresponding second image has been updated, that is, the current iteration calculation is not the first iteration calculation, then each grid in the corresponding second image is divided into two triangles to obtain multiple triangles. Based on the grid vertex position of the corresponding second image, the first triangle vertex position of each triangle is determined. Based on the grid vertex position of the corresponding second image, the first triangle vertex position of each triangle is determined. Then, based on the first triangle vertex position and the second triangle vertex position, the deformation error value of each triangle is determined. Based on the deformation error values ​​of all triangles, the regularization expression value of the corresponding second image is determined.

[0142] In this embodiment, the deformation error values ​​of the two triangles in each cell can be calculated according to the following formulas (8) and (9), wherein, referring to Figure 10 In formulas (8) and (9), This is the deformation error value. and Each grid consists of two triangles. and These represent the vertices of the triangle before and after the transformation. and These represent the side lengths of the triangle before and after the transformation:

[0143] Formula (8)

[0144] Formula (9)

[0145] Step 803: Determine the residual value based on the photometric error value and the regular expression value.

[0146] In this embodiment, a total cost function corresponding to the second image and the associated reference image is constructed based on the photometric error value and the regularization expression value, and the residue value is the value of the total cost function.

[0147] In one embodiment of this disclosure, referring to formula (10), wherein, in formula (10), and The grid vertex position can be optimized in advance based on the residual value f calculated in formula (10). For example, the grid vertex position is optimized for one pixel at a time. Then, the residual value is calculated based on the updated grid vertex position until the residual value meets the preset iteration stop condition.

[0148] Formula (10)

[0149] In the above calculation process, through experiments, it was found that setting the initial value of the mesh vertex to a 0 vector instead of the original mesh vertex position during the first iteration can improve alignment efficiency.

[0150] Therefore, referring to Figure 11 In the calculation of each image pair, the grid vertex position is optimized by iteratively calculating the residual value, and the iteratively calculated grid vertex position is passed to the next image pair. The next image pair performs further iterative calculation at the iteratively calculated grid vertex position, thereby improving the image alignment effect.

[0151] In summary, the image processing method of this disclosure can flexibly use different methods to determine the grid vertex position of the corresponding second image based on the corresponding second image and the associated reference image, thereby improving the accuracy of grid vertex position determination and thus enhancing the image alignment effect.

[0152] Based on the above embodiments, in order to further improve the image alignment efficiency, the image alignment can also be performed after downsampling the first image with higher resolution and the original reference image.

[0153] In one embodiment of this disclosure, such as Figure 12As shown, before generating multiple second images with different grid resolutions by dividing the first image into grids, the method further includes:

[0154] Step 1201: Determine whether the resolution of the first image is greater than a preset resolution threshold.

[0155] The preset resolution threshold is determined based on experimental data. Downsampling is not a necessary processing step. When the resolution of the first image is less than the preset resolution threshold, it is considered that the resolution of the original first image is already very low, and downsampling is performed to correct the error.

[0156] Step 1202: If the resolution is greater than the preset resolution threshold, the first image is downsampled according to the preset downsampling ratio to obtain the downsampled first image.

[0157] In this embodiment, if the resolution is greater than a preset resolution threshold, the first image is downsampled according to a preset downsampling ratio to obtain the downsampled first image.

[0158] For example, when the original resolution of the first image is The downsampling ratio is The first image is downsampled using bilinear interpolation, and the resolution of the downsampled first image is... ,in, , .

[0159] Step 1203: Downsample the original reference image according to the downsampling ratio to obtain the downsampled original reference image.

[0160] In this embodiment, for ease of processing, the original reference image is downsampled according to the downsampling ratio to obtain a downsampled original reference image. Then, image alignment is performed between the downsampled first image and the original reference image, improving the computational efficiency of the grid flow.

[0161] Therefore, in this embodiment, if the downsampling ratio is not 1, the acquired grid flow also needs to be upsampled, that is, the upsampling ratio corresponding to the preset downsampling ratio is determined. For example, the appropriate value for each point of the grid flow is... Then u and v can be respectively combined with the above and Multiplication is used to upsample the grid flow.

[0162] To enable those skilled in the art to have a more comprehensive understanding of the image processing method in this embodiment, an example based on a specific application scenario is provided below.

[0163] As shown in Figure 13(a), when multiple grids have a resolution of L, and the resolution of the first image is greater than a preset resolution threshold, the original first image and the original reference image are first downsampled. The downsampled first image is then divided into grids according to the grid resolution to obtain the second image. The size of the second image is scaled down to obtain a size-matched associated reference image to construct an image pair with an L-layer pyramid structure. Then, the grid vertex position of the grid-divided second image is used as the initial grid vertex position.

[0164] Propagate the grid vertex positions of the previous layer in order of increasing grid resolution, and perform alignment calculations on the image pairs of each layer. Construct residual values ​​based on the photometric error and regularization error between the corresponding second image and the associated reference image, and update the grid vertex positions of each layer based on the residual values.

[0165] Furthermore, the target mesh deformation image is obtained based on the mesh vertex positions of the maximum mesh resolution, and the mesh flow is calculated based on the target mesh deformation image and the first image. For example, as shown in Figure 13(b), taking pixel q as an example, its position in the mesh before deformation is determined by the vertices of the mesh before deformation. After updating the mesh vertex positions to control mesh deformation, the position of q is determined by the mesh vertex positions of the deformed mesh. The pixel q-grid flow is the movement vector from the pre-deformed grid to the post-deformed grid. This is to facilitate alignment of the original reference image and the first image according to the grid flow.

[0166] In summary, the image processing method of this disclosure, for a first image with a higher resolution, does not construct a grid in the original resolution image, but constructs a grid in the downsampled first image, thereby further improving image alignment efficiency.

[0167] Figure 14 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure. The apparatus can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 14 As shown, the device includes: an acquisition module 1410, an update module 1420, and an alignment calculation module 1430, wherein,

[0168] The acquisition module 1410 is used to acquire multiple sets of image pairs with different grid resolutions. Each set of image pairs includes a second image and an associated reference image. The second image is obtained by dividing the first image into grids at the corresponding grid resolution. The associated reference image is obtained by scaling the original reference image of the first image to be aligned according to the image size of the second image.

[0169] The update module 1420 is used to update the grid vertex positions of the corresponding second image based on the corresponding second image and the associated reference image when the current image pair has the smallest grid resolution, according to the grid resolution in ascending order.

[0170] The update module 1420 is also used to update the grid vertex position of the second image of the current image pair according to the grid vertex position of the previous image pair when the current image pair is not the minimum grid resolution, and to update the grid vertex position of the corresponding second image according to the corresponding second image and the associated reference image.

[0171] The alignment calculation module 1430 is used to obtain the target mesh deformation image based on the mesh vertex position of the second image corresponding to the current image pair when the current image pair has the maximum mesh resolution, and to calculate the mesh flow based on the target mesh deformation image and the first image, so as to align the original reference image and the first image according to the mesh flow.

[0172] The image processing apparatus provided in this disclosure can execute the image processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0173] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implements the image processing method described above.

[0174] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0175] The following is a detailed reference. Figure 15 The diagram illustrates a structural schematic suitable for implementing the electronic device 1500 in the embodiments of this disclosure. The electronic device 1500 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 15 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0176] like Figure 15As shown, electronic device 1500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1502 or a program loaded from storage device 1508 into random access memory (RAM) 1503. The RAM 1503 also stores various programs and data required for the operation of electronic device 1500. The processing device 1501, ROM 1502, and RAM 1503 are interconnected via bus 1504. Input / output (I / O) interface 1505 is also connected to bus 1504.

[0177] Typically, the following devices can be connected to I / O interface 1505: input devices 1506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1509. Communication device 1509 allows electronic device 1500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 15 An electronic device 1500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0178] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1509, or installed from storage device 1508, or installed from ROM 1502. When the computer program is executed by processing device 1501, it performs the functions defined in the aaaa method of embodiments of this disclosure.

[0179] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0180] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0181] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0182] The aforementioned computer-readable medium carries one or more programs. When these programs are executed by the electronic device, the electronic device causes the following: It acquires multiple sets of image pairs with different grid resolutions; calculates the grid vertex positions of the corresponding second image based on the second image and associated reference image for each set of image pairs in ascending order of grid resolution; and sequentially passes the grid vertex positions of the previous grid resolution to the next layer until the target grid deformed image corresponding to the image pair with the highest grid resolution is acquired; and calculates the grid flow based on the target grid deformed image and the first image to align the original reference image and the first image according to the grid flow. Thus, by controlling the grid shape deformation by passing the grid vertex positions of the previous image pair from coarse to fine according to the ascending order of grid resolution, and determining the grid flow based on the deformed grid, the efficiency and robustness of grid flow acquisition are improved, thereby enhancing the image alignment effect.

[0183] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0184] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0185] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0186] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0187] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0188] According to one or more embodiments of this disclosure, this disclosure provides an image processing method, including the following steps:

[0189] Multiple sets of image pairs with different grid resolutions are obtained, wherein each set of image pairs includes a second image and an associated reference image, wherein the second image is obtained by dividing the first image into grids at the corresponding grid resolution, and the associated reference image is obtained by scaling the original reference image of the first image to be aligned according to the image size of the second image;

[0190] Based on the order of grid resolution from smallest to largest, when the current image pair has the smallest grid resolution, the grid vertex positions of the corresponding second image are updated according to the corresponding second image and the associated reference image;

[0191] When the current image pair is not the minimum grid resolution, update the grid vertex position of the second image corresponding to the current image pair according to the grid vertex position of the previous image pair, and update the grid vertex position of the corresponding second image according to the corresponding second image and the associated reference image;

[0192] When the current image pair has the maximum grid resolution, the target grid deformation image is obtained according to the grid vertex position of the second image corresponding to the current image pair, and the grid flow is calculated according to the target grid deformation image and the first image, so as to align the original reference image and the first image according to the grid flow.

[0193] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, obtaining multiple sets of image pairs with different grid resolutions includes:

[0194] Based on the preset unit grid size and the multiple grid resolutions corresponding to the multiple sets of image pairs, the first image is divided into grids to generate multiple second images with different grid resolutions.

[0195] The original reference image is scaled according to the image size of the second image corresponding to each of the grid resolutions to generate a size-matched associated reference image;

[0196] The plurality of image pairs are generated based on the plurality of second images and the plurality of size-matched associated reference images.

[0197] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, determining the grid vertex position of the corresponding second image based on the corresponding second image and the associated reference image includes:

[0198] Based on the grid vertex positions of the corresponding second image, the residual values ​​between the corresponding second image and the associated reference image are calculated iteratively.

[0199] During each iteration, determine whether the residual value of the current iteration meets the preset iteration stopping condition;

[0200] If the iteration stopping condition is not met, the grid vertex position of the corresponding second image is updated according to the residual value, and the iteration calculation continues until the residual value of the current iteration calculation meets the preset iteration stopping condition.

[0201] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, the step of iteratively calculating the residual value between the corresponding second image and the associated reference image based on the grid vertex positions of the corresponding second image includes:

[0202] Based on the grid vertex positions of the corresponding second image, calculate the photometric error value between the corresponding second image and the associated reference image;

[0203] Calculate the regular expression value of the corresponding second image based on the grid vertex positions of the corresponding second image;

[0204] The residual value is determined based on the photometric error value and the regular expression value.

[0205] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, the step of calculating the photometric error value between the corresponding second image and the associated reference image based on the grid vertex positions of the corresponding second image includes:

[0206] Determine multiple sample pixels of the corresponding second image;

[0207] Determine the grid where each sample pixel is located, and determine the pixel position of each sample pixel based on the grid vertex position of the grid where it is located;

[0208] The photometric error value is determined based on a preset photometric error calculation algorithm and the pixel position.

[0209] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, determining a plurality of sample pixels of the corresponding second image includes:

[0210] The pixel gradient values ​​of each pixel in the second image at the current grid resolution are traversed one by one according to the preset step size;

[0211] Pixels whose pixel gradient values ​​are greater than or equal to the preset gradient threshold are identified as candidate sample pixels;

[0212] If the total number of candidate sample pixels is greater than a preset number threshold, then the candidate sample pixel of the preset number threshold is randomly determined as the sample pixel.

[0213] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, if the grid vertex positions of the corresponding second image have been updated, the step of calculating the regular expression value of the corresponding second image based on the grid vertex positions of the corresponding second image includes:

[0214] Each grid in the corresponding second image is divided into two triangles to obtain multiple triangles;

[0215] Based on the grid vertex positions of the corresponding second image, determine the first triangle vertex position of each triangle;

[0216] Obtain the position of the second vertex of each triangle in the corresponding second image before the most recent update;

[0217] The deformation error value of each triangle is determined based on the vertex positions of the first triangle and the vertex positions of the second triangle;

[0218] The regular expression value of the corresponding second image is determined based on the deformation error values ​​of all the triangles.

[0219] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, determining whether the residual value calculated in the current iteration meets a preset iteration stopping condition includes:

[0220] Determine whether the number of iterations in the current iteration is greater than or equal to the preset number of iterations for the current image pair, wherein the preset number of iterations is greater than 1;

[0221] If the number of iterations is greater than or equal to the preset number of iterations, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stopping condition.

[0222] According to one or more embodiments of this disclosure, the image processing method provided by this disclosure further includes:

[0223] If the number of iterations is not greater than or equal to the preset number of iterations, then the historical residual value of the previous iteration corresponding to the current image pair is obtained;

[0224] Determine whether the residual value calculated in the current iteration is greater than or equal to the historical residual value;

[0225] If the residual value is greater than or equal to the historical residual value, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stop condition.

[0226] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, updating the grid vertex position corresponding to the second image of the current image pair based on the grid vertex position of the previous image pair includes:

[0227] Calculate the grid resolution ratio between the current image pair and the previous image pair;

[0228] Determine the corresponding grid vertex of each grid vertex in the second image of the current image pair in the previous image pair;

[0229] The grid vertex position of each grid vertex in the second image of the current image pair is updated based on the product of the updated grid vertex position and the grid resolution ratio.

[0230] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, the step of calculating the mesh flow based on the target mesh deformation image and the first image includes:

[0231] Based on the positions of the grid vertices in the target mesh deformation image, obtain the first pixel position of each pixel in the target mesh deformation image;

[0232] Based on the initial grid vertex positions of the second image corresponding to the maximum grid resolution, the second pixel position of each pixel is obtained;

[0233] The mesh flow is obtained by calculating the displacement of the second pixel position and the first pixel position of each pixel.

[0234] According to one or more embodiments of this disclosure, the image processing method provided by this disclosure further includes, before generating a plurality of second images with different grid resolutions by dividing the first image into grids:

[0235] Determine whether the resolution of the first image is greater than a preset resolution threshold;

[0236] If the resolution is greater than the preset resolution threshold, the first image is downsampled according to the preset downsampling ratio to obtain the downsampled first image.

[0237] The original reference image is downsampled according to the downsampling ratio to obtain the downsampled original reference image.

[0238] According to one or more embodiments of this disclosure, the image processing method provided by this disclosure, before aligning the original reference image and the first image according to the grid flow, includes:

[0239] Determine the upsampling ratio corresponding to the preset downsampling ratio;

[0240] The upsampled grid flow is obtained by upsampling the grid flow according to the upsampling ratio.

[0241] According to one or more embodiments of the present disclosure, the present disclosure provides an image processing apparatus, comprising:

[0242] An acquisition module is used to acquire multiple sets of image pairs with different grid resolutions. Each set of image pairs includes a second image and an associated reference image. The second image is acquired by dividing the first image into grids at the corresponding grid resolution. The associated reference image is acquired by scaling the original reference image of the first image to be aligned according to the image size of the second image.

[0243] The update module is used to update the grid vertex positions of the corresponding second image according to the grid resolution in ascending order, when the current image pair has the smallest grid resolution, based on the corresponding second image and the associated reference image.

[0244] The update module is further configured to update the grid vertex position of the second image of the current image pair according to the grid vertex position of the previous image pair when the current image pair is not the minimum grid resolution, and update the grid vertex position of the corresponding second image according to the corresponding second image and the associated reference image.

[0245] The alignment calculation module is used to obtain a target mesh deformation image based on the mesh vertex position of the second image corresponding to the current image pair when the current image pair has the maximum mesh resolution, and to calculate a mesh flow based on the target mesh deformation image and the first image, so as to align the original reference image and the first image according to the mesh flow.

[0246] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the acquisition module is configured to:

[0247] Based on the preset unit grid size and the multiple grid resolutions corresponding to the multiple sets of image pairs, the first image is divided into grids to generate multiple second images with different grid resolutions.

[0248] The original reference image is scaled according to the image size of the second image corresponding to each of the grid resolutions to generate a size-matched associated reference image;

[0249] The plurality of image pairs are generated based on the plurality of second images and the plurality of size-matched associated reference images.

[0250] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the updating module includes:

[0251] The calculation unit is used to iteratively calculate the residual value between the corresponding second image and the associated reference image based on the grid vertex position of the corresponding second image;

[0252] The judgment unit is used to determine whether the residual value of the current iteration meets the preset iteration stop condition during each iteration calculation.

[0253] The update unit is used to update the grid vertex position of the corresponding second image according to the residual value when the iteration stopping condition is not met, and continue iterative calculation until the residual value of the current iteration calculation meets the preset iteration stopping condition.

[0254] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the computing unit is specifically used for:

[0255] Based on the grid vertex positions of the corresponding second image, calculate the photometric error value between the corresponding second image and the associated reference image;

[0256] Calculate the regular expression value of the corresponding second image based on the grid vertex positions of the corresponding second image;

[0257] The residual value is determined based on the photometric error value and the regular expression value.

[0258] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the computing unit is specifically used for:

[0259] Determine multiple sample pixels of the corresponding second image;

[0260] Determine the grid where each sample pixel is located, and determine the pixel position of each sample pixel based on the grid vertex position of the grid where it is located;

[0261] The photometric error value is determined based on a preset photometric error calculation algorithm and the pixel position.

[0262] According to one or more embodiments of the present disclosure, in the image processing apparatus provided by the present disclosure, the computing unit is specifically used to: traverse the pixel gradient value of each pixel in the second image at the current grid resolution one by one according to a preset step size;

[0263] Pixels whose pixel gradient values ​​are greater than or equal to the preset gradient threshold are identified as candidate sample pixels;

[0264] If the total number of candidate sample pixels is greater than a preset number threshold, then the candidate sample pixel of the preset number threshold is randomly determined as the sample pixel.

[0265] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, if the grid vertex positions of the corresponding second image have been updated, the calculation unit is specifically used for:

[0266] Each grid in the corresponding second image is divided into two triangles to obtain multiple triangles;

[0267] Based on the grid vertex positions of the corresponding second image, determine the first triangle vertex position of each triangle;

[0268] Obtain the position of the second vertex of each triangle in the corresponding second image before the most recent update;

[0269] The deformation error value of each triangle is determined based on the vertex positions of the first triangle and the vertex positions of the second triangle;

[0270] The regular expression value of the corresponding second image is determined based on the deformation error values ​​of all the triangles.

[0271] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the determining unit is specifically used for:

[0272] Determine whether the number of iterations in the current iteration is greater than or equal to the preset number of iterations for the current image pair, wherein the preset number of iterations is greater than 1;

[0273] If the number of iterations is greater than or equal to the preset number of iterations, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stopping condition.

[0274] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the determining unit is further configured to:

[0275] If the number of iterations is not greater than or equal to the preset number of iterations, then the historical residual value of the previous iteration corresponding to the current image pair is obtained;

[0276] Determine whether the residual value calculated in the current iteration is greater than or equal to the historical residual value;

[0277] If the residual value is greater than or equal to the historical residual value, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stop condition.

[0278] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the updating module is specifically used for:

[0279] Calculate the grid resolution ratio between the current image pair and the previous image pair;

[0280] Determine the corresponding grid vertex of each grid vertex in the second image of the current image pair in the previous image pair;

[0281] The grid vertex position of each grid vertex in the second image of the current image pair is updated based on the product of the updated grid vertex position and the grid resolution ratio.

[0282] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the alignment calculation module is specifically used for:

[0283] Based on the positions of the grid vertices in the target mesh deformation image, obtain the first pixel position of each pixel in the target mesh deformation image;

[0284] Based on the initial grid vertex positions of the second image corresponding to the maximum grid resolution, the second pixel position of each pixel is obtained;

[0285] The mesh flow is obtained by calculating the displacement of the second pixel position and the first pixel position of each pixel.

[0286] According to one or more embodiments of this disclosure, the image processing apparatus provided by this disclosure further includes: a downsampling module, used for:

[0287] Determine whether the resolution of the first image is greater than a preset resolution threshold;

[0288] If the resolution is greater than the preset resolution threshold, the first image is downsampled according to the preset downsampling ratio to obtain the downsampled first image.

[0289] The original reference image is downsampled according to the downsampling ratio to obtain the downsampled original reference image.

[0290] According to one or more embodiments of this disclosure, the image processing apparatus provided by this disclosure further includes: an upsampling module, used for:

[0291] Determine the upsampling ratio corresponding to the preset downsampling ratio;

[0292] The upsampled grid flow is obtained by upsampling the grid flow according to the upsampling ratio.

[0293] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0294] processor;

[0295] Memory used to store processor-executable instructions;

[0296] A processor is configured to read executable instructions from memory and execute the instructions to implement any of the image processing methods provided in this disclosure.

[0297] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium storing a computer program for performing any of the image processing methods provided in the present disclosure.

[0298] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0299] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0300] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image processing method, characterized in that, Includes the following steps: Obtain multiple pairs of images with different grid resolutions. Each image pair includes a second image and an associated reference image. The second image is obtained by dividing the first image into a grid at a corresponding grid resolution. The associated reference image is obtained by scaling the original reference image of the first image to be aligned according to the image size of the second image. According to the order of grid resolution from smallest to largest, when the current image pair has the smallest grid resolution, the grid vertex position of the corresponding second image is updated according to the corresponding second image and the associated reference image; When the current image pair is not the minimum grid resolution, update the grid vertex position of the second image corresponding to the current image pair according to the grid vertex position of the previous image pair, and update the grid vertex position of the corresponding second image according to the corresponding second image and the associated reference image; When the current image pair has the maximum grid resolution, the target grid deformation image is obtained according to the grid vertex position of the second image corresponding to the current image pair, and the grid flow is calculated according to the target grid deformation image and the first image, so as to align the original reference image and the first image according to the grid flow.

2. The method as described in claim 1, characterized in that, The acquisition of multiple pairs of images with different grid resolutions includes: Based on the preset unit grid size and the multiple grid resolutions corresponding to the multiple sets of image pairs, the first image is divided into grids to generate multiple second images with different grid resolutions. The original reference image is scaled according to the image size of the second image corresponding to each of the grid resolutions to generate a size-matched associated reference image; The plurality of image pairs are generated based on the plurality of second images and the plurality of size-matched associated reference images.

3. The method as described in claim 1, characterized in that, The step of updating the mesh vertex positions of the corresponding second image based on the corresponding second image and the associated reference image includes: Based on the grid vertex positions of the corresponding second image, the residual values ​​between the corresponding second image and the associated reference image are calculated iteratively. During each iteration, determine whether the residual value of the current iteration meets the preset iteration stopping condition; If the iteration stopping condition is not met, the grid vertex position of the corresponding second image is updated according to the residual value, and the iteration calculation continues until the residual value of the current iteration calculation meets the preset iteration stopping condition.

4. The method as described in claim 3, characterized in that, The step of iteratively calculating the residual value between the corresponding second image and the associated reference image based on the grid vertex positions of the corresponding second image includes: Based on the grid vertex positions of the corresponding second image, calculate the photometric error value between the corresponding second image and the associated reference image; Calculate the regular expression value of the corresponding second image based on the grid vertex positions of the corresponding second image; The residual value is determined based on the photometric error value and the regular expression value.

5. The method as described in claim 4, characterized in that, The step of calculating the photometric error value between the corresponding second image and the associated reference image based on the grid vertex positions of the corresponding second image includes: Determine multiple sample pixels of the corresponding second image; Determine the grid where each sample pixel is located, and determine the pixel position of each sample pixel based on the grid vertex position of the grid where it is located; The photometric error value is determined based on a preset photometric error calculation algorithm and the pixel position.

6. The method as described in claim 5, characterized in that, Determining the multiple sample pixels of the corresponding second image includes: The pixel gradient values ​​of each pixel in the second image at the current grid resolution are traversed one by one according to the preset step size; Pixels whose pixel gradient values ​​are greater than or equal to the preset gradient threshold are identified as candidate sample pixels; If the total number of candidate sample pixels is greater than a preset threshold, then the preset threshold number of candidate sample pixels are randomly selected as the sample pixels.

7. The method as described in claim 4, characterized in that, If the grid vertex positions of the corresponding second image have been updated, then calculating the regular expression value of the corresponding second image based on the grid vertex positions of the corresponding second image includes: Each grid in the corresponding second image is divided into two triangles to obtain multiple triangles; Based on the grid vertex positions of the corresponding second image, determine the first triangle vertex position of each triangle; Obtain the position of the second vertex of each triangle in the corresponding second image before the most recent update; The deformation error value of each triangle is determined based on the vertex positions of the first triangle and the vertex positions of the second triangle; The regular expression value of the corresponding second image is determined based on the deformation error values ​​of all the triangles.

8. The method as described in claim 3, characterized in that, The step of determining whether the residual value calculated in the current iteration meets the preset iteration stopping condition includes: Determine whether the number of iterations in the current iteration is greater than or equal to the preset number of iterations for the current image pair, wherein the preset number of iterations is greater than 1; If the number of iterations is greater than or equal to the preset number of iterations, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stopping condition.

9. The method as described in claim 8, characterized in that, Also includes: If the number of iterations is not greater than or equal to the preset number of iterations, then the historical residual value of the previous iteration corresponding to the current image pair is obtained; Determine whether the residual value calculated in the current iteration is greater than or equal to the historical residual value; If the residual value is greater than or equal to the historical residual value, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stop condition.

10. The method as described in claim 1, characterized in that, The step of updating the grid vertex position of the second image in the current image pair according to the grid vertex position of the previous image pair includes: Calculate the grid resolution ratio between the current image pair and the previous image pair; Determine the corresponding grid vertex of each grid vertex in the second image of the current image pair in the previous image pair; The grid vertex position of each grid vertex in the second image of the current image pair is updated based on the product of the updated grid vertex position and the grid resolution ratio.

11. The method as described in claim 1, characterized in that, The step of calculating the mesh flow based on the target mesh deformation image and the first image includes: Based on the positions of the grid vertices in the target mesh deformation image, obtain the first pixel position of each pixel in the target mesh deformation image; Based on the initial grid vertex positions of the second image corresponding to the maximum grid resolution, the second pixel position of each pixel is obtained; The grid flow is obtained by calculating the displacement between the second pixel position and the first pixel position of each pixel.

12. The method as described in claim 2, characterized in that, Before generating multiple second images with different grid resolutions by dividing the first image into grids, the method further includes: Determine whether the resolution of the first image is greater than a preset resolution threshold; If the resolution is greater than the preset resolution threshold, the first image is downsampled according to the preset downsampling ratio to obtain the downsampled first image. The original reference image is downsampled according to the downsampling ratio to obtain the downsampled original reference image.

13. The method as described in claim 12, characterized in that, Before aligning the original reference image and the first image according to the grid flow, the process includes: Determine the upsampling ratio corresponding to the preset downsampling ratio; The upsampled grid flow is obtained by upsampling the grid flow according to the upsampling ratio.

14. An image processing apparatus, characterized in that, include: An acquisition module is used to acquire multiple sets of image pairs with different grid resolutions. Each set of image pairs includes a second image and an associated reference image. The second image is acquired by dividing the first image into grids at the corresponding grid resolution. The associated reference image is acquired by scaling the original reference image of the first image to be aligned according to the image size of the second image. The update module is used to update the grid vertex positions of the corresponding second image according to the grid resolution in ascending order, when the current image pair has the smallest grid resolution, based on the corresponding second image and the associated reference image. The update module is further configured to update the grid vertex position of the second image of the current image pair according to the grid vertex position of the previous image pair when the current image pair is not the minimum grid resolution, and update the grid vertex position of the corresponding second image according to the corresponding second image and the associated reference image. The alignment calculation module is used to obtain a target mesh deformation image based on the mesh vertex position of the second image corresponding to the current image pair when the current image pair has the maximum mesh resolution, and to calculate a mesh flow based on the target mesh deformation image and the first image, so as to align the original reference image and the first image according to the mesh flow.

15. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image processing method according to any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the image processing method according to any one of claims 1-13.

Images