Image processing method, apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to an image processing method, device, equipment and medium, wherein the method comprises: calculating an optical flow of a first image and an original reference image to be aligned; determining a position change state of the first image relative to an image background; determining a candidate flow field corresponding to the position change state; and aligning the first image and the original reference image according to the optical flow and the candidate flow field. Thus, the image alignment is performed in combination with the characteristics of the optical flow and the candidate flow field, and the image alignment effect can be guaranteed regardless of whether the relevant object in the first image is relatively static or dynamic relative to the original reference image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and medium. Background Technology

[0002] Optical flow is an image alignment technique. Generally, optical flow is caused by the movement of foreground objects in a scene, camera movement, or both. When the human eye observes a moving object, the image of the object forms a series of continuously changing images on the retina. This series of continuously changing information "flows" across the retina (i.e., the image plane), like a "flow" of light, hence the name optical flow. Optical flow expresses changes in an image, and because it contains information about the motion of objects, it is often used in image processing to determine the motion of targets.

[0003] In related technologies, optical flow calculations utilize the temporal changes of pixels in an image sequence and the correlation between the target image and the reference image to be aligned to find the motion vector field of objects between the two images, thus obtaining the optical flow. When calculating the optical flow for image alignment, the two-dimensional velocity vector of each pixel in the reference image moving to its corresponding pixel in the target image is calculated. The optical flow is then obtained based on the two-dimensional velocity vectors of all pixels, and the alignment between the images is achieved based on the optical flow.

[0004] However, the above-mentioned alignment between images based on optical flow relies on obtaining the two-dimensional velocity vector of each pixel. When there are areas with indistinct textures or objects that are not moving in the image, the two-dimensional velocity vector of the corresponding pixel cannot be obtained, which affects the accuracy of optical flow acquisition and, consequently, the alignment effect of the image. Summary of the Invention

[0005] To address or at least partially address the aforementioned technical problems, this disclosure provides an image processing method, the method comprising: calculating the optical flow of a first image and an original reference image to be aligned; determining the positional change state of the first image relative to the image background; determining a candidate flow field corresponding to the positional change state; and aligning the first image and the original reference image according to the optical flow and the candidate flow field.

[0006] This disclosure also provides an image processing apparatus, the apparatus comprising: a calculation module for calculating the optical flow of a first image and an original reference image to be aligned; a first determination module for determining the positional change state of the first image relative to the image background; a second determination module for determining a candidate flow field corresponding to the positional change state; and an image alignment module for aligning the first image and the original reference image according to the optical flow and the candidate flow field.

[0007] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the image processing method provided in this disclosure.

[0008] This disclosure also provides a computer-readable storage medium storing a computer program for performing the image processing method provided in this disclosure.

[0009] The technical solution provided in this disclosure has the following advantages compared with the prior art: The image processing solution provided in this disclosure calculates the optical flow between a first image and an original reference image to be aligned, then determines the positional change state of the first image relative to the image background, determines the candidate flow field corresponding to the positional change state, and finally aligns the first image and the original reference image based on the optical flow and the candidate flow field. Therefore, by combining the characteristics of the candidate flow field and the optical flow for image alignment, the image alignment effect can be guaranteed regardless of whether the relevant objects in the first image are relatively static or dynamic relative to the original reference image. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of the present disclosure;

[0012] Figure 2 A schematic diagram illustrating the display of optical flow in a color model according to an embodiment of this disclosure;

[0013] Figure 3 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0014] Figure 4 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0015] Figure 5 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0016] Figure 6(a) is a scene diagram of an image pair provided in an embodiment of this disclosure;

[0017] Figure 6(b) is a scene diagram of another image pair provided in an embodiment of this disclosure;

[0018] Figure 7(a) is a scene diagram of another image pair provided in an embodiment of this disclosure;

[0019] Figure 7(b) is a schematic diagram of another image pair generation scenario provided by an embodiment of this disclosure;

[0020] Figure 8 This is a schematic diagram of an image alignment scenario provided by an embodiment of the present disclosure;

[0021] Figure 9 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0022] Figure 10 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0023] Figure 11 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0024] Figure 12 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0025] Figure 13 This is a schematic diagram of a mesh deformation scene provided in an embodiment of the present disclosure;

[0026] Figure 14 This is a schematic diagram of a mesh similarity transformation scenario provided by an embodiment of the present disclosure;

[0027] Figure 15 This is a schematic diagram of another image alignment scenario provided by an embodiment of the present disclosure;

[0028] Figure 16 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0029] Figure 17 A schematic flowchart of another image processing method provided in an embodiment of this disclosure;

[0030] Figure 18 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure;

[0031] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0038] To address the aforementioned issues, this disclosure provides an image processing method that combines the characteristics of mesh flow and optical flow for image alignment. Mesh flow is an image alignment technique that involves dividing the image to be aligned into a checkerboard pattern and then deforming the image by moving the positions of the grid vertices. Ultimately, the deformed target image is aligned to a reference image. Since mesh flow moves the pixels within the grid by moving the grid vertices, image alignment is achieved through grid deformation. The position of each pixel is determined by the coordinates of its grid vertex, eliminating the need to identify the specific location of each pixel in the image. Therefore, it provides better tracking performance for areas with limited texture or static objects, compensating for the shortcomings of optical flow tracking. Consequently, regardless of whether the objects in the first image are relatively static or dynamic relative to the original reference image, the image alignment effect can be guaranteed.

[0039] The method will be described below with reference to specific embodiments.

[0040] Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of the present disclosure. The method can be executed by an image processing device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes:

[0041] Step 101: Calculate the optical flow between the first image and the original reference image to be aligned.

[0042] In this embodiment, the purpose of image processing is to align the first image and the original reference image. The original reference image and the first image can be adjacent video frames in the video stream, or non-adjacent video frames in the same video stream, etc. They can be taken by the same camera or by different cameras, etc., and there are no restrictions here.

[0043] As mentioned above, the optical flow of the first image and the original reference image to be aligned is actually composed of the motion field formed by the moving velocity vectors of the pixels in the first image aligning to the pixels in the original reference image. Therefore, in one embodiment of this disclosure, the first pixel position of each pixel in the first image is determined, the second pixel position of each pixel in the original reference image is determined, and then the instantaneous velocity vector of the second pixel position moving to the first pixel position is calculated. The optical flow is obtained based on the instantaneous velocity vectors of all pixels in the first image.

[0044] In actual execution, the algorithm for calculating the optical flow formed by the instantaneous velocity vector can be the DenseInverse Search-based method, the Lucas-Kanade method, etc., and no restrictions are imposed here.

[0045] Step 102: Determine the positional change of the first image relative to the image background.

[0046] It should be understood that optical flow should be a reflection of the motion field of objects in the first image relative to the original reference image. This motion field is caused by the movement of objects within the first image itself, for example, as... Figure 2 As shown (where, for ease of understanding, Figure 2 The image shows a first image overlaid in pixel dimension. When viewing optical flow in HSV (Hue, Saturation, Value) color, the depth of the color spot represents the motion rate (the image distinguishes the depth of color by gray value). White represents the area where the object is almost motionless. Based on optical flow, the motion field formed by the instantaneous velocity vector of the person moving in the first image can be obtained.

[0047] However, when capturing the first image and the original reference image to be aligned, the position of the camera relative to the image background may be in motion. This motion will also be reflected in the optical flow between the first image and the original reference image. This optical flow is obviously not caused by the movement of objects in the first image. Therefore, in order to avoid this situation, in this embodiment, the position change state of the first image relative to the image background is determined. This position change state is used to indicate whether the camera has moved relative to the image background of the first image when capturing the first image. When there is movement, the position change state of the first image relative to the image background is considered to be in motion. When there is no movement, the position change state of the first image relative to the image background is considered to be in a stationary state.

[0048] Step 103: Determine the candidate flow field corresponding to the position change state.

[0049] In this embodiment, a candidate flow field corresponding to the position change state is determined. This candidate flow field is used to compensate for the optical flow error caused by the position change motion state. If the position change state is a motion state, it indicates that the instantaneous velocity vector in the optical flow obtained by aligning the first image with the original reference image to be aligned may be affected by the camera motion. Therefore, the candidate optical flow field can be a flow field such as a grid flow that is good at tracking static backgrounds. If the position change state is a stationary state, it indicates that the instantaneous velocity vector in the optical flow obtained by aligning the first image with the original reference image to be aligned reflects the motion of the target object itself. Therefore, the candidate optical flow field can be a zero optical flow field (a zero optical flow field limits this embodiment to image alignment based solely on optical flow that is good at tracking dynamic motion), or it can be any other flow field that is good at tracking object motion.

[0050] It should be noted that, for ease of explanation, in the following embodiments, the position change state is taken as the motion state, the candidate optical flow field is taken as the grid flow, and the position change state is taken as the stationary state, the candidate optical flow field is taken as the zero optical flow field.

[0051] In some possible embodiments, if the position change state is a motion state, it indicates that the instantaneous velocity vector in the optical flow obtained by the alignment calculation of the first image and the original reference image to be aligned may be affected by the camera motion. Therefore, the grid flow of the first image and the original reference image is calculated, and the grid flow is determined as a candidate flow field. As mentioned above, the grid flow is good at tracking static backgrounds, etc. Therefore, if the object in the first image itself may not be moving, the alignment calculation can be performed based on the grid flow.

[0052] Of course, in one embodiment of this disclosure, if the position change state is a stationary state, it indicates that the instantaneous velocity vector in the optical flow obtained by the alignment calculation of the first image and the original reference image to be aligned reflects the motion of the first target object itself. Therefore, at this time, the image alignment process is directly performed based on the optical flow calculated above. In practical applications, the zero optical flow field is determined as the candidate flow field.

[0053] Step 104: Align the first image and the original reference image based on optical flow and candidate flow fields.

[0054] In this embodiment, the first image and the original reference image are aligned based on optical flow and candidate flow fields. For example, a target flow with a small alignment error is identified in the optical flow and candidate flow fields, and image alignment is performed based on this target flow. Thus, by combining the characteristics of the candidate flow fields and optical flow, image alignment can be performed regardless of whether the relevant objects in the first image are relatively static or dynamic relative to the original reference image. Aligning the original reference image and the first image here can be achieved by denoising the first image based on the target flow, etc. Of course, in some scenarios, a better synthesis effect can also be achieved by synthesizing the denoised first image and the original reference image, avoiding the synthesis of "ghosting".

[0055] In summary, the image processing method of this disclosure calculates the optical flow between a first image and an original reference image to be aligned, then determines the positional change state of the first image relative to the image background, determines the candidate flow field corresponding to the positional change state, and finally aligns the first image and the original reference image based on the optical flow and the candidate flow field. Therefore, by combining the characteristics of the candidate flow field and the optical flow for image alignment, the image alignment effect can be guaranteed regardless of whether the relevant objects in the first image are relatively static or dynamic relative to the original reference image.

[0056] It should be noted that the method for determining the positional change of the first image relative to the image background differs in different application scenarios. Examples are provided below:

[0057] In one embodiment of this disclosure, such as Figure 3 As shown, determining the positional change state of the first image relative to the image background includes:

[0058] Step 301: Determine the instantaneous velocity vector of each sample pixel in the first image based on optical flow.

[0059] Among them, sample pixels can be selected based on gradient, that is, pixels in the first image with gradient values ​​greater than a certain value are determined as sample pixels, or they can be selected based on grayscale values ​​and other parameters, without any restrictions.

[0060] If the optical flow is calculated based on the granularity of the optical flow block, then the instantaneous velocity vector of each sample pixel can be the instantaneous velocity vector of the optical flow block to which it belongs.

[0061] Step 302: Determine the number of pixels whose instantaneous velocity vector magnitude is less than a preset velocity threshold.

[0062] The preset speed threshold can be calibrated based on experimental data. In some possible embodiments, the preset speed threshold can be a value between 0.25 and 0.5.

[0063] Step 303: Calculate the ratio of the number of pixels to the total number of sample pixels in the first image.

[0064] Step 304: If the ratio is greater than the preset ratio threshold, then the position change state is determined to be a motion state.

[0065] Step 305: If the ratio is less than or equal to the preset ratio threshold, then the position change state is determined to be a stationary state.

[0066] The preset ratio threshold can be calibrated based on experimental data. In some possible embodiments, the preset ratio threshold can be a value between 0.15 and 0.3.

[0067] In this embodiment, since optical flow algorithms typically restrict the first image and the original reference image to be captured in close proximity with minimal object movement, such as adjacent image frames, if the ratio is greater than a preset ratio threshold, it indicates that the displacement of the pixels in the first image relative to the original reference image is relatively small, which conforms to the constraints of the optical flow algorithm. At the same time, it is also clear that there is almost no displacement caused by camera pose movement. Therefore, it is determined that the state of position change is a stationary state.

[0068] If the ratio is not greater than the preset ratio threshold, it means that the pixel in the first image has a relatively large displacement relative to the original reference image, which obviously does not meet the constraints of the optical flow algorithm. It can be assumed that this large displacement is caused by the movement of the camera during shooting. Therefore, the positional change state of the first image and the original reference image is considered to be a motion state.

[0069] In one embodiment of this disclosure, such as Figure 4 As shown, determining the positional change state of the first image relative to the image background includes:

[0070] Step 401: Obtain the first camera pose of the first image and the second camera pose of the second image.

[0071] Step 402: If the poses of the first camera and the second camera are inconsistent, then the position change state is determined to be a motion state.

[0072] Step 403: If the poses of the first camera and the second camera are the same, then the position change state is determined to be a stationary state.

[0073] Among them, camera pose can be understood as the shooting posture and position of the camera, including the camera's intrinsic parameters and extrinsic parameters. The camera's extrinsic parameters include the camera's shooting world coordinates.

[0074] It is easy to understand that, since optical flow algorithms usually restrict the first image and the original reference image to be image frames that were captured recently and whose object movement is minimal, when the camera pose of the first image and the original reference image are the same, the optical flow of the captured first image and the original reference image is hardly affected by the camera movement.

[0075] Therefore, in this embodiment, it is determined whether the pose of the first camera and the pose of the second camera are consistent. If they are consistent, the position change state is considered to be a stationary state; otherwise, if they are inconsistent, the position change state is considered to be a moving state.

[0076] In summary, the image processing method of this disclosure can flexibly determine the positional change state of the first image relative to the image background, providing a reference for selecting a suitable flow field for image alignment.

[0077] In related technologies, feature point matching is performed based on mesh flow technology. That is, after determining the matching feature points between the image to be aligned and the reference image, the displacement of the mesh vertices is calculated based on the displacement of the feature points. Then, the mesh vertex positions of the image to be aligned are moved by the displacement to achieve image alignment.

[0078] However, the above-mentioned image alignment method based on feature points requires the detection and matching of feature points, which results in a long processing time and affects the efficiency of image alignment. Furthermore, feature points in image regions with weak local textures are difficult to measure, which leads to large errors in the calculation of mesh vertex displacement, and thus affects the quality of image alignment.

[0079] Therefore, in the embodiments of this disclosure, to handle large displacements between the image to be aligned and the reference image, multiple grid resolutions of a multi-layer pyramid structure are constructed. Different numbers of grid divisions are set for each layer of the image according to the multiple grid resolutions. The optimization results corresponding to other grid resolutions are passed from coarse to fine, and then to the image corresponding to the maximum grid resolution, resulting in the final determined grid flow. This ensures the efficiency and robustness of the grid flow determination, thereby improving the image alignment effect.

[0080] The process of calculating the grid flow between the first image and the original reference image is explained below with reference to specific embodiments.

[0081] like Figure 5 As shown, calculating the grid flow between the first image and the original reference image includes:

[0082] Step 501: Obtain multiple sets of image pairs with different grid resolutions based on the first image and the original reference image.

[0083] Each image pair includes a second image and an associated reference image. The second image is obtained by dividing the first image into a grid at the corresponding grid resolution, and the associated reference image is obtained by scaling the original reference image according to the image size of the second image.

[0084] In one embodiment of this disclosure, a first image can be divided into multiple second images with different grid resolutions according to a preset unit grid size and multiple grid resolutions.

[0085] In this embodiment, the preset unit grid size is usually smaller than the first image size, equivalent to the size of one grid cell in the grid image after grid division. The preset unit grid size can be calibrated based on experimental data.

[0086] To handle situations where there is a large pixel displacement, such as a large displacement between the original reference image and the first image, multiple grid resolutions can be preset, and the first image can be divided into grids to generate multiple second images with different grid resolutions.

[0087] As shown in Figure 6(a), multiple grid resolutions can be arranged in a pyramid structure, meaning that after arranging the corresponding second images according to size, the ratio of grid resolutions in adjacent second images is the same. In some possible embodiments, as shown in Figure 6(b), after arranging the corresponding second images according to ascending order of grid resolutions, the ratio between adjacent grid resolutions can also be different.

[0088] In this embodiment, the first image is first divided into grids to generate multiple second images with different grid resolutions. The number of unit grids in each second image is consistent with the corresponding grid resolution. For example, as shown in Figure 7(a), when multiple grid resolutions correspond to three grid resolutions, namely “1*1”, “2*2” and “4*4”, the corresponding second images contain chessboard structures with 1 unit grid, 4 unit grids and 16 unit grids, respectively.

[0089] Then, the original reference image is scaled according to the image size of the second image corresponding to each grid resolution to generate a size-matched associated reference image.

[0090] In this embodiment, in order to achieve image alignment at various coarse and fine granularities and to take into account various displacement degrees for subsequent alignment processing, the original reference image is scaled according to the image size of the second image corresponding to each grid resolution to generate a size-matched associated reference image. It can also be understood that the second image corresponding to each grid resolution has an associated reference image of the same size. The associated reference image is only consistent with the size of the corresponding second image, and the associated reference image does not need to be gridded.

[0091] Continuing with the scenario shown in Figure 7(a), as shown in Figure 7(b), the three second images correspond to three associated reference images of the same size. Therefore, for each second image corresponding to a grid resolution, an associated reference image of the same size is set to be matched with it. The second image and the associated reference image at the same grid resolution form an image pair, thereby improving the computational efficiency of image alignment.

[0092] Step 502: According to the order of grid resolution from smallest to largest, when the current image pair has the smallest grid resolution, update the grid vertex position of the corresponding second image based on the corresponding second image and the associated reference image.

[0093] In this embodiment, it should be understood that the alignment of the second image and the associated reference image can be achieved by displacing the grid vertices to deform the grid in the second image. The grid deformation causes the displacement of the pixels in the second image, thereby achieving alignment between the second image after grid deformation and the associated reference image at the corresponding grid resolution.

[0094] Taking a second image with a grid resolution of 1*1 as an example, the second image contains a grid A1, whose corresponding associated reference image is B. Then, for A1, the coordinate vectors of its four grid vertices are respectively... and Each pixel within grid A1 can be represented by the coordinate vector of the vertex of its grid. Taking pixel q as an example, the coordinate vector of the pixel's position can be expressed as follows: (1), where C in the following formula... k The weights for each grid vertex can be preset:

[0095]

[0096] Furthermore, when aligning the second image with the corresponding associated reference image, the coordinate vectors of the determined grid vertices are as follows: and After the grid A1 is transformed into A2 by the control of the grid vertex, taking the pixel q' in A2 as an example, the coordinate vector of the pixel q' can be expressed as the following formula (2):

[0097]

[0098] Therefore, it can be understood that in this embodiment, by aligning the second image and the corresponding associated reference image, the positions of the grid vertices of the second image are updated. The updated grid vertex positions are the deformed grid vertices. Therefore, the deformation of the grid in the second image can obviously be controlled according to the deformation of the grid vertices. The deformed grid drives the movement of the pixels inside the grid, which actually realizes the movement and alignment of the pixels inside the grid. Thus, in this embodiment, there is no need to detect and calculate the displacement of the feature points between the second image to be aligned and the associated reference image, which greatly improves the alignment efficiency.

[0099] In this embodiment, since the calculation is performed in order of increasing resolution and in order of coarse to fine displacement granularity, the purpose of the calculation is to obtain the result of the mesh deformation of the second image with the maximum mesh resolution. Therefore, in this embodiment, if the current image pair has the minimum mesh resolution, the mesh vertex position of the corresponding second image can be updated simply based on the corresponding second image and the associated reference image, without needing to control the actual deformation of the second image.

[0100] The updated grid vertex position can be understood as the position of the network vertex of the associated reference image after the second image is aligned with the displacement granularity at the corresponding grid resolution. The grid vertex position can be understood as the coordinate vector of the grid vertex, etc.

[0101] Step 503: When the current image pair is not the minimum grid resolution, update the grid vertex position of the second image corresponding to the current image pair according to the grid vertex position of the previous image pair, and update the grid vertex position of the corresponding second image according to the corresponding second image and the associated reference image.

[0102] In this embodiment, the grid vertex positions of the second image in the current image pair are updated according to the grid vertex positions of the previous image pair, in order of increasing grid resolution. That is, the aligned grid vertex positions at each grid resolution are passed sequentially. The grid vertex positions correspond to the deformation of the grid shape at each grid resolution. Therefore, in this embodiment, the deformed grid shape is passed to the last second image in order of increasing grid resolution.

[0103] In this embodiment, if the current image pair is not at the minimum grid resolution, the grid vertex position of the second image corresponding to the current image pair is updated according to the grid vertex position of the previous image pair. The grid vertex position of the previous image pair is the position of the already aligned coarser-grained grid vertex after deformation. Therefore, by combining the grid vertex positions of the previous image pair, the deformation of the previously aligned coarser-grained grid shape is transferred to the second image at the current grid resolution, further improving the image alignment efficiency.

[0104] In actual execution, in order to facilitate the transmission of the grid vertex positions of the previous image pair, it is also necessary to determine the grid vertex positions of the previous image pair corresponding to the grid vertices in the current image pair.

[0105] In this embodiment, to determine the position of the corresponding grid vertex in the second image of the current image pair, the grid resolution ratio between the current image pair and the previous image pair can be calculated. This resolution ratio is used to indicate the coordinate ratio of each grid vertex in the adjacent image pair. Then, after determining the corresponding grid vertex of each grid vertex in the second image of the current image pair in the previous image pair, since the resolution ratio is used to indicate the coordinate ratio of each grid vertex in the adjacent image pair, the grid vertex position of each grid vertex in the second image of the current image pair is updated according to the product of the updated grid vertex position of the corresponding grid vertex and the grid resolution ratio.

[0106] However, in practical applications, determining the position of each grid vertex after the grid vertex update may require obtaining the weighted values ​​of the three other grid vertices belonging to the same unit grid. In order to ensure the stability of grid deformation, the weighted values ​​of the grid vertices are the same as the weighted values ​​after the grid vertex positions are not updated. Therefore, in one embodiment of this disclosure, the original grid vertex positions of the grid vertices of the second image in the previous image pair before the grid vertex positions are updated can also be determined according to the grid resolution ratio. The weighted values ​​between the grid vertices are then calculated based on these vertex positions to facilitate the subsequent determination of the position of each grid vertex after the update.

[0107] To facilitate understanding, the following example illustrates the process of updating the grid vertex position of the second image in the current image pair based on the grid vertex position of the previous image pair, using a ratio of 2 in both the horizontal and vertical coordinates.

[0108] In this embodiment, when multiple grid resolutions of multiple image pairs correspond to L (L is 3 in this embodiment) grid resolutions, namely "1*1", "2*2" and "4*4", the second images and associated reference images between multiple image pairs can be arranged in order of increasing grid resolution to construct an L-layer image pyramid structure.

[0109] If the current image corresponds to layer L of the pyramid structure, where L is greater than 1, then propagate the mesh shape of layer L-1 to the current layer. Let Mesh... L This represents the grid corresponding to the second image of layer L. This represents the original grid corresponding to the second image in layer L-1. This represents the deformed mesh corresponding to the second image of layer L-1 (this deformed mesh can refer to the mesh after updating the mesh vertex positions based on the second image of layer L-1 and the associated reference image).

[0110] Traversing Mesh L Each grid vertex V in L Since the resolution of the lower grid is twice that of the upper grid in adjacent grid resolutions, the grid vertex V is obtained according to formula (3). L exist The coordinates V of the corresponding grid vertex L-1 :

[0111] V L-1 =0.5V L Formula (3)

[0112] Furthermore, V can be obtained based on bilinear interpolation algorithms, etc. L-1 exist Interpolation information in V to obtain VL-1 The four grid vertices of the corresponding cell and the weight w of each grid vertex k (k = 1, 2, 3, 4), V L-1 It can be represented as a weighted linear combination of the four vertices, as shown in formula (4);

[0113]

[0114] Furthermore, after obtaining the weight of each grid vertex, the weight is calculated according to formula (5). Corresponding mesh vertices in, for exist The corresponding mesh vertices:

[0115]

[0116] Furthermore, the position V of the original mesh vertex corresponding to the second image of the current image pair is determined according to formula (6). L Update the mesh vertex positions based on the mesh vertex positions of the previous image pair.

[0117]

[0118] This cycle is repeated to achieve the transfer of deformable mesh shape from coarse to fine.

[0119] For example, such as Figure 8 As shown, when multiple grid resolutions correspond to L (L is 3 in this embodiment) grid resolutions, namely "1*1", "2*2" and "4*4", a pyramid structure layer of the second image corresponding to L grid resolutions can be constructed in ascending order. First, the grid vertex positions of the second image in the first layer are determined based on the first layer second image and the associated reference image. Then, the initial grid vertex positions of the second image in the second layer are updated based on the grid vertex positions of the two images. The grid vertex positions of the second image in the second layer are determined for the updated second image and the associated reference image in the second layer. At this time, the grid vertex positions are the result of the combined effect of the alignment of the first layer grid vertex and the alignment of the second layer grid vertex.

[0120] Similarly, the initial mesh vertex positions of the second image in the third layer are updated based on the mesh vertex positions of the second image in the second layer. The mesh vertex positions of the second image in the third layer are then determined using the updated second image and the associated reference image in the third layer. At this point, the mesh vertex positions of the second image in the third layer are the result of the combined effects of mesh vertex alignment in the second and third layers. This achieves the transfer of mesh deformation shapes from the first layer to the second and third layers.

[0121] Step 504: When the current image pair has the maximum grid resolution, obtain the target grid deformation image based on the grid vertex position of the second image corresponding to the current image pair, and calculate the grid flow based on the target grid deformation image and the first image, so as to align the original reference image and the first image according to the grid flow.

[0122] As mentioned above, in this embodiment, the sensitivity of the determined grid vertex positions to the degree of displacement varies depending on the grid resolution. Therefore, in order to integrate displacements of multiple granularities for image alignment, in this embodiment, the second image corresponding to the maximum grid resolution is adjusted according to the alignment displacement information of the grid vertices corresponding to each grid resolution to obtain the target grid deformation image. That is, the displacement of the grid vertices under all displacement degrees is ultimately reflected in the grid shape at the maximum grid resolution in this embodiment. In other words, based on the grid vertex positions of the current image and the corresponding second image, the displacement of each pixel within the grid can be known, and thus, the target grid deformation image can be obtained.

[0123] In this embodiment, since the grid flow is good at tracking static backgrounds, the grid flow mainly reflects the motion field of the background pixels between the original reference image and the first image. In this embodiment, the grid flow can be calculated based on the target grid deformation image and the first image. That is, the motion field corresponding to the deformation of the first image to the target grid deformation image is the grid flow. Then, the original reference image and the first image are aligned according to the grid flow. The alignment of the original reference image and the first image can be: controlling the deformation of the first image according to the grid flow, or denoising the first image according to the grid flow, etc. Of course, in some scenarios, a better synthesis effect can also be achieved by compositing the denoised first image and the original reference image to avoid synthesizing "ghosting".

[0124] It should be noted that the method for calculating the mesh flow based on the target mesh deformation image and the first image differs in different application scenarios:

[0125] In some possible embodiments, a deep learning model is learned in advance using deep learning technology, and the corresponding mesh flow is output based on the input target mesh deformation image and the first image using the deep learning model.

[0126] In some possible embodiments, such as Figure 9 As shown, the mesh flow is calculated based on the target mesh deformation image and the first image, including:

[0127] Step 901: Based on the positions of the grid vertices in the target mesh deformation image, obtain the third pixel position of each pixel in the target mesh deformation image.

[0128] As mentioned above, the pixel position of each pixel can be represented by the vertex coordinates of the grid it belongs to. Therefore, by determining the grid vertex positions in the target grid deformation image, the third pixel position of each pixel in the target grid deformation image can be obtained.

[0129] Step 902: Obtain the fourth pixel position of each pixel based on the initial grid vertex position of the second image corresponding to the maximum grid resolution.

[0130] The original second image here can be understood as the second image corresponding to the grid resolution when the grid was initially divided.

[0131] In this embodiment, the fourth pixel position of each pixel in the original second image is obtained based on the initial grid vertex position of the second image corresponding to the maximum grid resolution. Similarly, the fourth pixel position of each pixel can be determined based on the vertex coordinates of the grid vertex where the pixel is located in the original second image.

[0132] Step 903: Calculate the displacement of the fourth pixel position and the corresponding third pixel position of each pixel to obtain the mesh flow.

[0133] In fact, the grid flow can be regarded as the displacement motion field of pixels during the image alignment process. Therefore, in this embodiment, the displacement difference between the fourth pixel position and the corresponding third pixel position of each pixel is calculated to obtain the grid flow.

[0134] In summary, the image processing method of this disclosure acquires multiple sets of image pairs with different grid resolutions. Based on the grid resolution from smallest to largest, it sequentially calculates the grid vertex positions of the corresponding second image and associated reference image for each set of image pairs. The grid vertex positions of the previous grid resolution are then passed to the next layer until the target grid deformed image corresponding to the image pair with the largest grid resolution is obtained. A grid flow is then calculated based on the target grid deformed image and the first image to facilitate alignment of the original reference image and the first image. Therefore, by controlling the grid shape deformation by passing the grid vertex positions of the previous image pair from coarse to fine according to the grid resolution from smallest to largest, and determining the grid flow based on the deformed grid, the efficiency and robustness of grid flow acquisition are improved, thus enhancing image alignment performance.

[0135] In practical applications, the mesh vertex positions of the second image and the associated reference image in each image pair are calculated, thereby achieving image alignment at the corresponding displacement granularity. The method for determining the mesh vertex positions of the corresponding second image based on the corresponding second image and the associated reference image varies in different application scenarios, as shown in the following examples:

[0136] In one embodiment of this disclosure, a deep learning model is pre-trained based on a large amount of sample data. The input of the deep learning model is two images to be aligned, and the output is the grid vertex position of the target image to be aligned. Therefore, in this embodiment, the corresponding second image and the associated reference image are input into the pre-trained deep learning model to obtain the grid vertex position of the corresponding second image.

[0137] In another embodiment of this disclosure, in order to further improve the image alignment effect, the grid vertex positions of the second image are iteratively calculated for each pair of images.

[0138] like Figure 10 As shown, the mesh vertex positions of the corresponding second image are determined based on the corresponding second image and the associated reference image, including:

[0139] Step 1001: Based on the grid vertex positions of the corresponding second image, iteratively calculate the residual values ​​between the corresponding second image and the associated reference image.

[0140] The residual value here can be understood as the vertex alignment error between the second image and the associated reference image. The grid vertex position of the corresponding second image is updated based on this alignment error to achieve the gradual alignment between the second image and the associated reference image.

[0141] In different application scenarios, the residual value includes different contents. In some possible embodiments, the residual value includes photometric error value, that is, it includes the brightness error of the pixels of the second image and the associated reference image, thereby determining the vertex alignment error of the second image and the associated reference image based on the brightness dimension.

[0142] In some other possible embodiments, the residual value includes a deformation error value that ensures that the squares within the mesh conform to the similarity transformation as much as possible when deformed, thereby determining the vertex alignment error between the second image and the associated reference image based on the shape dimension.

[0143] In some other possible embodiments, the vertex alignment error between the second image and the associated reference image can be determined by combining the aforementioned photometric error value and deformation error value.

[0144] Step 1002: During each iteration, determine whether the residual value of the current iteration meets the preset iteration stop condition.

[0145] In this embodiment, to avoid slow image alignment efficiency due to too many iterations, an iteration stop condition is set. Once the iteration stop condition is met, the iteration calculation of the current image pair is stopped, and the alignment calculation of the next set of image pairs begins.

[0146] The preset iteration stopping condition varies in different application scenarios:

[0147] In one embodiment of this disclosure, a preset number of iterations matching the grid resolution is set for each image pair. Iteration stops when the preset number of iterations is greater than or equal to this preset number. For example, when the grid resolution is 1*1, the preset number of iterations can be between 5 and 10; when the grid resolution is 2*2, the preset number of iterations can be between 3 and 5; and when the grid resolution is 4*24, the preset number of iterations can be between 5 and 10.

[0148] In another embodiment of this disclosure, the iteration stopping condition is determined based on the iteration performance. That is, in this embodiment, it is determined whether the residual value of the iteration calculation is greater than or equal to the residual value of the previous iteration calculation. If it is greater than or equal to the residual value of the previous iteration calculation, it means that the iteration performance has deteriorated. Therefore, it is determined that the iteration stopping condition is met and the iteration calculation is stopped.

[0149] In another embodiment of this disclosure, in order to balance iteration efficiency and iteration performance, the iteration stopping condition is determined by combining the above two embodiments.

[0150] In this embodiment, as Figure 11 As shown, determining whether the residual value calculated in the current iteration meets the preset iteration stopping condition includes:

[0151] Step 1101: Determine whether the number of iterations calculated in the current iteration is greater than or equal to the preset number of iterations for the current image pair. The preset number of iterations is greater than 1.

[0152] Step 1102: If the number of iterations is greater than or equal to the preset number of iterations, then the residual value calculated in the current iteration is determined to meet the preset iteration stopping condition.

[0153] In this embodiment, it is assumed that the current iteration count is iter. l The preset number of iterations is If iter l Greater than or equal to Then it is considered that the preset iteration stopping condition is met.

[0154] Step 1103: If the number of iterations is not greater than or equal to the preset number of iterations, then obtain the historical residual value of the previous iteration for the current image pair.

[0155] Step 1104: Determine whether the residual value calculated in the current iteration is greater than or equal to the historical residual value.

[0156] Step 1105: If the residual value is greater than or equal to the historical residual value, then the residual value calculated in the current iteration is determined to meet the preset iteration stop condition.

[0157] In this embodiment, if the number of iterations is not greater than or equal to a preset number of iterations, the historical residual value of the previous iteration corresponding to the current image pair is obtained. If the residual value of the current iteration is greater than or equal to the historical residual value, it is determined that the performance of the current iteration has decreased, thereby determining that the residual value of the current iteration satisfies the preset iteration stopping condition.

[0158] For example, when the historical residual value is r iter-1 The residual value of the current image pair is r iter Then when r iter Greater than or equal to r iter-1 When the residual value of the current iteration is satisfied with the preset iteration stopping condition, it is determined that the residual value is satisfied with the current iteration.

[0159] Step 1003: If the iteration stopping condition is not met, update the grid vertex position of the corresponding second image according to the residual value, and continue iterative calculation until the residual value of the current iteration calculation meets the preset iteration stopping condition.

[0160] In this embodiment, since the residual value is the alignment error, the grid vertex position of the corresponding second image is updated according to the residual value, and the iteration calculation continues until the residual value calculated in the current iteration meets the preset iteration stop condition.

[0161] Of course, in this embodiment, iterative calculation is not necessary. Instead, the residual value between the corresponding second image and the associated reference image can be calculated only once. In this case, the grid vertex position of the second image is updated based on the residual value, according to the original image of the second image.

[0162] To enable those skilled in the art to more clearly understand how to iteratively calculate the residual values ​​of the corresponding second image and the associated reference image based on the grid vertex positions of the corresponding second image, the following explanation will focus on the residual values ​​including the aforementioned photometric error values ​​and deformation error values.

[0163] In one embodiment of this disclosure, such as Figure 12 As shown, based on the grid vertex positions of the corresponding second image, the residual values ​​between the corresponding second image and the associated reference image are iteratively calculated, including:

[0164] Step 1201: Calculate the photometric error value between the corresponding second image and the associated reference image based on the grid vertex position of the corresponding second image.

[0165] In this embodiment, the photometric error value can be determined based on the brightness difference between all pixels in the second image and their corresponding pixels in the associated reference image. However, when there are a large number of pixels, the computational load may be large, affecting the image alignment efficiency.

[0166] Therefore, in one embodiment of this disclosure, multiple sample pixels in the second image are determined, and the image alignment efficiency is improved by reducing the number of pixels used to calculate the photometric error value.

[0167] In this embodiment, firstly, multiple sample pixels of the corresponding second image are determined, and then, the grid in which each sample pixel is located is determined. The pixel position of each sample pixel is determined according to the grid vertex position of the grid in which it is located (as mentioned in the example above, the position of a pixel can be obtained by weighting the grid vertex position of its cell).

[0168] The sample pixels mentioned above can be randomly determined or selected based on brightness values. In this embodiment, in order to further reduce the number of pixels used to calculate the photometric error value without affecting the image alignment effect, a uniform random sampling technique is introduced to further reduce the number of sample pixels used to construct the photometric error value.

[0169] In this embodiment, the pixel gradient value of each pixel in the second image at the current grid resolution is traversed one by one according to a preset step size. Pixels with pixel gradient values ​​greater than or equal to a preset gradient threshold are determined as candidate sample pixels. That is, if the gradient of the second image at the current grid resolution is divided into horizontal gradients... and gradient in the vertical direction Then, if the gradient value of pixel q in the horizontal direction is abs(q) x ) and the gradient value abs(q) in the numerical direction y The sum of ) is greater than or equal to the preset gradient threshold t g If the gradient threshold is set, the corresponding pixel is determined as a candidate sample pixel. In some possible embodiments, the preset gradient threshold can be between 0.05 and 2.

[0170] In one embodiment of this disclosure, in order to further improve the sample collection efficiency, after determining the number of candidate samples, the total number of candidate sample pixels can also be determined. When the total number is greater than or equal to the total number, the determination of candidate sample pixels is stopped, and all candidate sample pixels are used as sample pixels.

[0171] Alternatively, in some other possible embodiments, if the total number of candidate sample pixels is greater than a preset threshold (for example, the total number can be between 800 and 1000), the candidate sample pixels of the preset threshold are randomly determined as the sample pixels. For example, after randomly arranging all the candidate sample pixels, the candidate sample pixels belonging to the preset threshold in the random arrangement result are determined as multiple sample pixels. Thus, this random sampling strategy can significantly reduce the sample pixel size while ensuring a uniform distribution of samples on the image, improving computational efficiency while maintaining tracking performance.

[0172] Furthermore, after determining the sample pixels, the pixel position of each sample pixel is determined based on the grid where each sample pixel is located and the position of the grid vertex. The photometric error value is then determined based on the preset photometric error calculation algorithm and the pixel position.

[0173] It should be noted that the preset photometric error calculation algorithm in this embodiment can be any existing method for calculating photometric error, and no limitation is imposed here.

[0174] For example, the preset photometric error calculation algorithm is formula (7), where, in formula (7), f q I is the photometric error value. tar (q) and I tar (q) represents the brightness value of sample pixel q in the second image and the associated reference image, respectively. Let be the gradient of the sample pixel q in the second image, and let the four grid vertices of the cell containing q be... (k=1,2,3,4),C k The weights for each grid vertex can be preset.

[0175]

[0176] In this embodiment, as Figure 13 As shown, the mesh deformation process before and after the mesh vertex position update (where solid quadrilaterals represent the cell mesh before the update and dashed quadrilaterals represent the cell mesh after the update) is illustrated. Mesh deformation is achieved through vertex displacement, and the interpolation coefficient C of q before and after deformation is shown. k While remaining unchanged, the mesh deformation carries the position of q to a more suitable position q', thereby reducing the photometric error.

[0177] Step 1202: Calculate the regular expression value of the corresponding second image based on the grid vertex position of the corresponding second image.

[0178] Among them, the regular expression value can be regarded as the deformation error value to ensure that the squares in the grid conform to the similarity transformation as much as possible when they are deformed. Based on the regular expression value, the image filled in the foreground area of ​​the aligned second image can look more natural and more in line with the perspective transformation.

[0179] In this embodiment, if the grid vertex position of the corresponding second image has been updated, that is, the current iteration calculation is not the first iteration calculation, then each grid in the corresponding second image is divided into two triangles to obtain multiple triangles. Based on the grid vertex position of the corresponding second image, the first triangle vertex position of each triangle is determined. Based on the grid vertex position of the corresponding second image, the first triangle vertex position of each triangle is determined. Then, based on the first triangle vertex position and the second triangle vertex position, the deformation error value of each triangle is determined. Based on the deformation error values ​​of all triangles, the regularization expression value of the corresponding second image is determined.

[0180] In this embodiment, the deformation error values ​​of the two triangles in each cell can be calculated according to the following formulas (8) and (9), wherein, referring to Figure 14 In formulas (8) and (9), f d The deformation error value is represented by tri1 and tri2, which are the two triangles included in each grid, respectively. k {k = 1, 2, 3} and {V'} k {k = 1, 2, 3} represent the vertices of the triangle before and after the transformation, respectively. k1k2 ,k1k2=12,213,23} and {l' k1k2 The numbers k1k2 = 12, 213, 23 represent the side lengths of the triangle before and after the transformation, respectively.

[0181]

[0182]

[0183] Step 1203: Determine the residual value based on the photometric error value and the regular expression value.

[0184] In this embodiment, a total cost function corresponding to the second image and the associated reference image is constructed based on the photometric error value and the regularization expression value, and the residue value is the value of the total cost function.

[0185] In one embodiment of this disclosure, referring to formula (10), where λ1 and λ2 can be preset, the grid vertex position is optimized according to the residual value f calculated in formula (10), for example, the grid vertex position of one pixel is optimized each time, and then the residual value is calculated according to the updated grid vertex position until the residual value meets the preset iteration stop condition.

[0186]

[0187] In the above calculation process, through experiments, it was found that setting the initial value of the mesh vertex to a 0 vector instead of the original mesh vertex position during the first iteration can improve alignment efficiency.

[0188] Therefore, referring to Figure 15 In the calculation of each image pair, the residual value is calculated iteratively to optimize the position of the grid vertex and then passed to the next image pair. The next image pair performs further iterative calculations at the calculated grid vertex positions, thereby improving the image alignment effect.

[0189] In summary, the image processing method of this disclosure can flexibly use different methods to determine the grid vertex position of the corresponding second image based on the corresponding second image and the associated reference image, thereby improving the accuracy of grid vertex position determination and thus enhancing the image alignment effect.

[0190] Based on the above embodiments, in order to further improve the image alignment efficiency, the image alignment can also be performed after downsampling the first image with higher resolution and the original reference image.

[0191] In one embodiment of this disclosure, such as Figure 16 As shown, before generating multiple second images with different grid resolutions by dividing the first image into grids, the method further includes:

[0192] Step 1601: Determine whether the resolution of the first image is greater than a preset resolution threshold.

[0193] The preset resolution threshold is determined based on experimental data. Downsampling is not a necessary processing step. When the resolution of the first image is less than the preset resolution threshold, it is considered that the resolution of the original first image is already very low, and downsampling is performed to correct the error.

[0194] Step 1602: If the resolution is greater than the preset resolution threshold, the first image is downsampled according to the preset downsampling ratio to obtain the downsampled first image.

[0195] In this embodiment, if the resolution is greater than a preset resolution threshold, the first image is downsampled according to a preset downsampling ratio to obtain the downsampled first image.

[0196] For example, when the original resolution of the first image is W ori ×H ori The downsampling ratio is (α) x ,α y The first image is downsampled using bilinear interpolation, and the resolution of the downsampled first image is W×H, where α x =W ori / W, α y =H ori / H.

[0197] Step 1603: Downsample the original reference image according to the downsampling ratio to obtain the downsampled original reference image.

[0198] In this embodiment, for ease of processing, the original reference image is downsampled according to the downsampling ratio to obtain a downsampled original reference image. Then, image alignment is performed between the downsampled first image and the original reference image, improving the computational efficiency of the grid flow.

[0199] Therefore, in this embodiment, if the downsampling ratio is not 1, the acquired grid flow also needs to be upsampled, that is, the upsampling ratio corresponding to the preset downsampling ratio needs to be determined. For example, the appropriate value for each point of the grid flow is flow. p =(u,v) T Then u and v can be respectively combined with the above α x and α y Multiplication is used to upsample the grid flow.

[0200] In summary, the image processing method of this disclosure, for a first image with a higher resolution, does not construct a grid in the original resolution image, but constructs a grid in the downsampled first image, which further improves the image alignment efficiency based on grid flow.

[0201] Based on the above embodiments, in order to ensure the alignment effect, the target flow with better alignment effect is selected from the mesh flow and optical flow.

[0202] In one embodiment of this disclosure, such as Figure 17 As shown, aligning the first image and the original reference image according to optical flow and mesh flow includes:

[0203] Step 1701: Determine the third image that aligns the first image with the original reference image based on the optical flow.

[0204] In some possible embodiments, a third image that aligns the first image with the original reference image can be calculated based on bilinear interpolation according to optical flow, wherein the third image is the result of aligning the first image with the original reference image in the optical flow dimension.

[0205] Step 1702: Determine the fourth image that aligns the first image with the original reference image based on the candidate flow field.

[0206] In some possible embodiments, a fourth image that aligns the first image with the original reference image can be calculated based on bilinear interpolation according to the candidate optical flow field, wherein the fourth image is the result of aligning the first image with the original reference image in the grid flow dimension.

[0207] Step 1703: Calculate the first photometric error image between the third image and the original reference image, and calculate the second photometric error image between the fourth image and the original reference image.

[0208] In this embodiment, a first photometric error image can be calculated between the third image and the original reference image. This involves calculating the photometric error value of each sample pixel in the third image within the third image and within the original reference image. The first photometric error image is determined based on the absolute value of the photometric error. The photometric error value can be calculated using any existing photometric error calculation algorithm, and no limitation is imposed here. Similarly, a second photometric error image can be calculated between the fourth image and the original reference image.

[0209] In some possible embodiments, the first photometric error image and the second photometric error image are determined according to the following formulas (11) and (12), wherein, in formulas (11) and (12), This is the first photometric error image. For the second photometric error image, I ref For the original reference image, For the third image, This is the fourth image. `abs` represents the absolute value of the photometric error. This represents the first photometric error image calculated between the third image and the original reference image. This represents the second photometric error image calculated between the fourth image and the original reference image:

[0210]

[0211]

[0212] Step 1704: Determine the target flow in the optical flow and candidate flow field based on the first photometric error image and the second photometric error image, so as to align the first image and the original reference image according to the target flow.

[0213] In this embodiment, the first photometric error image reflects the alignment effect of the third image with the original reference image, and the second photometric error image reflects the alignment effect of the fourth image with the original reference image. Since the third image is aligned based on optical flow and the fourth image is aligned based on candidate flow field, the target flow can be determined in optical flow and candidate flow field based on the first photometric error image and the second photometric error image, so as to align the first image and the original reference image according to the target flow.

[0214] In one embodiment of this disclosure, to further improve the reliability of determining the target flow, multiple superpixel blocks can be obtained by superpixel segmentation of the original reference image using superpixel extraction algorithms such as Superpixels Extracted via Energy-Driven Sampling. The first photometric error mean of the multiple superpixel blocks is determined based on a first photometric error image. Similarly, the second photometric error mean of the multiple superpixel blocks is determined based on a second photometric error image, and the first photometric error mean is then determined. Is it greater than the mean of the second photometric error? If the optical flow is greater than the second mean photometric error, the candidate flow field is determined to be the target flow; otherwise, if it is not greater than the second mean photometric error, the optical flow is determined to be the target flow.

[0215] In actual judgment, the optical flow of the first image calculated based on the optical flow algorithm may have a guiding bias. For example, the aforementioned DIS algorithm tends to obtain results with lower photometric errors, which may affect the fairness of judging whether the mean of the first photometric error is greater than the mean of the second photometric error. Therefore, in order to balance this guiding bias, in one embodiment of this disclosure, a preset relaxation parameter value σ corresponding to the optical flow is determined. σ is related to the optical flow calculation algorithm. When the optical flow calculation algorithm is DIS, σ can be a number between 0.5 and 1.0. Then, the preset relaxation parameter value and the mean of the first photometric error are summed, and the mean of the first photometric error is updated according to the summation result, that is, the optical flow of the first image calculated based on the first photometric error is determined. The mean of the first photometric error used for comparison, if Then the optical flow is determined to be the target flow. Then the candidate flow field is determined as the target flow, etc.

[0216] In another embodiment of this disclosure, the alignment error of the first image with the original reference image based on optical flow can be calculated based on the gradient value dimension, and the alignment error of the first image with the original reference image based on the candidate flow field dimension, to determine the target flow with a smaller alignment error.

[0217] In summary, the image processing method disclosed herein determines the alignment effect of the first image with the original reference image based on optical flow and candidate flow fields, and selects the target flow with better alignment effect to align the first image and the original reference image. Regardless of whether the relevant objects in the first image are relatively static or dynamic relative to the original reference image, the image alignment effect can be guaranteed.

[0218] Figure 18 This is a schematic diagram of an image processing apparatus according to an embodiment of the present disclosure. The apparatus can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 18As shown, the device includes: a calculation module 1810, a first determination module 1820, a second determination module 1830, and an image alignment module 1840, wherein,

[0219] The calculation module 1810 is used to calculate the optical flow between the first image and the original reference image to be aligned;

[0220] The first determining module 1820 is used to determine the positional change state of the first image relative to the image background;

[0221] The second determining module 1830 is used to determine the candidate flow field corresponding to the position change state;

[0222] Image alignment module 1840 is used to align the first image and the original reference image based on optical flow and candidate flow fields.

[0223] The image processing apparatus provided in this disclosure can execute the image processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0224] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implement the image processing method described above.

[0225] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.

[0226] The following is a detailed reference. Figure 19 The diagram illustrates a structural schematic suitable for implementing the electronic device 1900 in the embodiments of this disclosure. The electronic device 1900 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 19 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0227] like Figure 19As shown, electronic device 1900 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1901, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1902 or a program loaded from storage device 1908 into random access memory (RAM) 1903. RAM 1903 also stores various programs and data required for the operation of electronic device 1900. Processing unit 1901, ROM 1902, and RAM 1903 are interconnected via bus 1904. Input / output (I / O) interface 1905 is also connected to bus 1904.

[0228] Typically, the following devices can be connected to the I / O interface 1905: input devices 1906 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1907 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1908 including, for example, magnetic tape, hard disk, etc.; and communication devices 1909. Communication device 1909 allows electronic device 1900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 19 An electronic device 1900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0229] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1909, or installed from storage device 1908, or installed from ROM 1902. When the computer program is executed by processing device 1901, it performs the functions defined in the image processing method of embodiments of this disclosure.

[0230] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0231] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0232] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0233] The aforementioned computer-readable medium carries one or more programs. When these programs are executed by the electronic device, the electronic device: calculates the optical flow between a first image and an original reference image to be aligned; then, determines the positional change state of the first image relative to the image background; determines a candidate flow field corresponding to the positional change state; and finally, aligns the first image and the original reference image based on the optical flow and the candidate flow field. Thus, by combining the characteristics of the candidate flow field and the optical flow for image alignment, the image alignment effect can be guaranteed regardless of whether the relevant objects in the first image are relatively static or dynamic relative to the original reference image.

[0234] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0235] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0236] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0237] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0238] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0239] According to one or more embodiments of this disclosure, this disclosure provides an image processing method, including the following steps:

[0240] Calculate the optical flow between the first image and the original reference image to be aligned;

[0241] Determine the positional change state of the first image relative to the image background;

[0242] Determine the candidate flow field corresponding to the position change state;

[0243] Align the first image and the original reference image based on the optical flow and the candidate flow field.

[0244] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, calculating the optical flow between the first image and the original reference image to be aligned includes:

[0245] Determine the first pixel position of each pixel in the first image, and determine the second pixel position of each pixel in the original reference image;

[0246] Calculate the instantaneous velocity vector as the second pixel position moves to the first pixel position;

[0247] The optical flow is obtained based on the instantaneous velocity vector of all pixels in the first image.

[0248] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, determining the positional change state of the first image relative to the image background includes:

[0249] The instantaneous velocity vector of each sample pixel in the first image is determined based on the optical flow;

[0250] Determine the number of pixels whose magnitude of the instantaneous velocity vector is less than a preset velocity threshold;

[0251] If the ratio is greater than a preset ratio threshold, then the position change state is determined to be a motion state;

[0252] If the ratio is less than or equal to the preset ratio threshold, then the position change state is determined to be a stationary state.

[0253] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, determining the positional change state of the first image relative to the image background includes:

[0254] Obtain the first camera pose of the first image and the second camera pose of the second image;

[0255] If the poses of the first camera and the poses of the second camera are inconsistent, then the position change state is determined to be a motion state.

[0256] If the pose of the first camera and the pose of the second camera are the same, then the position change state is determined to be a stationary state.

[0257] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, determining the candidate flow field corresponding to the position change state includes:

[0258] If the position change state is a motion state, then calculate the grid flow between the first image and the original reference image, and determine the grid flow as the candidate flow field;

[0259] If the position change state is a stationary state, then the zero-optical flow field is determined as the candidate flow field.

[0260] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, calculating the grid flow between the first image and the original reference image includes:

[0261] Multiple sets of image pairs with different grid resolutions are obtained based on the first image and the original reference image.

[0262] Each image pair includes a second image and an associated reference image. The second image is obtained by dividing the first image into grids at a corresponding grid resolution. The associated reference image is obtained by scaling the original reference image according to the image size of the second image.

[0263] According to the order of grid resolution from smallest to largest, when the current image pair has the smallest grid resolution, the grid vertex position of the corresponding second image is updated according to the corresponding second image and the associated reference image;

[0264] When the current image pair is not the minimum grid resolution, update the grid vertex position of the second image corresponding to the current image pair according to the grid vertex position of the previous image pair, and update the grid vertex position of the corresponding second image according to the corresponding second image and the associated reference image;

[0265] When the current image pair has the maximum grid resolution, the target grid deformation image is obtained based on the grid vertex position of the second image corresponding to the current image pair, and the grid flow is calculated based on the target grid deformation image and the first image.

[0266] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, obtaining multiple sets of image pairs with different grid resolutions based on the first image and the original reference image includes:

[0267] Based on the preset unit grid size and the multiple grid resolutions corresponding to the multiple sets of image pairs, the first image is divided into grids to generate multiple second images with different grid resolutions.

[0268] The original reference image is scaled according to the image size of the second image corresponding to each of the grid resolutions to generate a size-matched associated reference image;

[0269] The plurality of image pairs are generated based on the plurality of second images and the plurality of size-matched associated reference images.

[0270] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, updating the mesh vertex positions of the corresponding second image based on the corresponding second image and the associated reference image includes:

[0271] Based on the grid vertex positions of the corresponding second image, the residual values ​​between the corresponding second image and the associated reference image are calculated iteratively.

[0272] During each iteration, determine whether the residual value of the current iteration meets the preset iteration stopping condition;

[0273] If the iteration stopping condition is not met, the grid vertex position of the corresponding second image is updated according to the residual value, and the iteration calculation continues until the residual value of the current iteration calculation meets the preset iteration stopping condition.

[0274] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, the step of iteratively calculating the residual value between the corresponding second image and the associated reference image based on the grid vertex positions of the corresponding second image includes:

[0275] Based on the grid vertex positions of the corresponding second image, calculate the photometric error value between the corresponding second image and the associated reference image;

[0276] Calculate the regular expression value of the corresponding second image based on the grid vertex positions of the corresponding second image;

[0277] The residual value is determined based on the photometric error value and the regular expression value.

[0278] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, the step of calculating the photometric error value between the corresponding second image and the associated reference image based on the grid vertex positions of the corresponding second image includes:

[0279] Determine multiple sample pixels of the corresponding second image;

[0280] Determine the grid where each sample pixel is located, and determine the pixel position of each sample pixel based on the grid vertex position of the grid where it is located;

[0281] The photometric error value is determined based on a preset photometric error calculation algorithm and the pixel position of the sample pixel.

[0282] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, determining a plurality of sample pixels of the corresponding second image includes:

[0283] The pixel gradient values ​​of each pixel in the second image at the current grid resolution are traversed one by one according to the preset step size;

[0284] Pixels whose pixel gradient values ​​are greater than or equal to the preset gradient threshold are identified as candidate sample pixels;

[0285] If the total number of candidate sample pixels is greater than a preset threshold, then the preset threshold number of candidate sample pixels are randomly selected as the sample pixels.

[0286] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, if the grid vertex positions of the corresponding second image have been updated, the step of calculating the regular expression value of the corresponding second image based on the grid vertex positions of the corresponding second image includes:

[0287] Each grid in the corresponding second image is divided into two triangles to obtain multiple triangles;

[0288] Based on the grid vertex positions of the corresponding second image, determine the first triangle vertex position of each triangle;

[0289] Obtain the position of the second vertex of each triangle in the corresponding second image before the most recent update;

[0290] The deformation error value of each triangle is determined based on the vertex positions of the first triangle and the vertex positions of the second triangle;

[0291] The regular expression value of the corresponding second image is determined based on the deformation error values ​​of all the triangles.

[0292] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, determining whether the residual value calculated in the current iteration meets a preset iteration stopping condition includes:

[0293] Determine whether the number of iterations in the current iteration is greater than or equal to the preset number of iterations for the current image pair, wherein the preset number of iterations is greater than 1;

[0294] If the number of iterations is greater than or equal to the preset number of iterations, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stopping condition.

[0295] According to one or more embodiments of this disclosure, the image processing method provided by this disclosure further includes:

[0296] If the number of iterations is not greater than or equal to the preset number of iterations, then the historical residual value of the previous iteration corresponding to the current image pair is obtained;

[0297] Determine whether the residual value calculated in the current iteration is greater than or equal to the historical residual value;

[0298] If the residual value is greater than or equal to the historical residual value, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stop condition.

[0299] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, the step of calculating the mesh flow based on the target mesh deformation image and the first image includes:

[0300] Based on the positions of the grid vertices in the target mesh deformation image, obtain the third pixel position of each pixel in the target mesh deformation image;

[0301] Based on the initial grid vertex positions of the second image corresponding to the maximum grid resolution, obtain the fourth pixel position of each pixel;

[0302] The grid flow is obtained by calculating the displacements of the third and fourth pixel positions of each pixel.

[0303] According to one or more embodiments of this disclosure, the image processing method provided by this disclosure, before generating a plurality of second images with different grid resolutions by dividing the first image into grids, further includes:

[0304] Determine whether the resolution of the first image is greater than a preset resolution threshold;

[0305] If the resolution is greater than the preset resolution threshold, the first image is downsampled according to the preset downsampling ratio to obtain the downsampled first image.

[0306] The original reference image is downsampled according to the downsampling ratio to obtain the downsampled original reference image.

[0307] According to one or more embodiments of this disclosure, the image processing method provided by this disclosure, before aligning the first image and the original reference image according to the optical flow and the mesh flow, includes:

[0308] Determine the upsampling ratio corresponding to the preset downsampling ratio;

[0309] The upsampled grid flow is obtained by upsampling the grid flow according to the upsampling ratio.

[0310] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, aligning the first image and the original reference image based on the optical flow and the candidate flow field includes:

[0311] The first image is aligned with the third image of the original reference image based on the optical flow.

[0312] Based on the candidate flow field, a fourth image is determined that aligns the first image with the original reference image;

[0313] Calculate the first photometric error image between the third image and the original reference image, and calculate the second photometric error image between the fourth image and the original reference image;

[0314] Based on the first photometric error image and the second photometric error image, a target flow is determined in the optical flow and the candidate flow field to facilitate alignment of the first image and the original reference image according to the target flow.

[0315] According to one or more embodiments of this disclosure, in the image processing method provided by this disclosure, determining the target flow in the optical flow and the candidate flow field based on the first photometric error image and the second photometric error image includes:

[0316] The original reference image is segmented into multiple superpixel blocks;

[0317] The first photometric error mean of the plurality of superpixel blocks is determined based on the first photometric error image;

[0318] The mean value of the second photometric error of the plurality of superpixel blocks is determined based on the second photometric error image;

[0319] Determine whether the mean value of the first photometric error is greater than the mean value of the second photometric error;

[0320] If the error is greater than the second mean photometric error, then the candidate flow field is determined to be the target flow.

[0321] If the optical flow is not greater than the second mean photometric error, then the optical flow is determined to be the target flow.

[0322] According to one or more embodiments of this disclosure, the image processing method provided by this disclosure further includes, before determining whether the mean value of the first photometric error is greater than the mean value of the second photometric error:

[0323] Determine the preset relaxation parameter value corresponding to the optical flow;

[0324] The preset relaxation parameter value and the mean value of the first photometric error are summed, and the mean value of the first photometric error is updated based on the summation result.

[0325] According to one or more embodiments of the present disclosure, the present disclosure provides an image processing apparatus, comprising:

[0326] A calculation module is used to calculate the optical flow between the first image and the original reference image to be aligned;

[0327] The first determining module is used to determine the positional change state of the first image relative to the image background;

[0328] The second determining module is used to determine the candidate flow field corresponding to the position change state;

[0329] An image alignment module is used to align the first image and the original reference image according to the optical flow and the candidate flow field. According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the calculation module is specifically used for:

[0330] Determine the first pixel position of each pixel in the first image, and determine the second pixel position of each pixel in the original reference image;

[0331] Calculate the instantaneous velocity vector as the second pixel position moves to the first pixel position;

[0332] The optical flow is obtained based on the instantaneous velocity vector of all pixels in the first image.

[0333] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the first determining module is specifically used for:

[0334] The instantaneous velocity vector of each sample pixel in the first image is determined based on the optical flow;

[0335] Determine the number of pixels whose magnitude of the instantaneous velocity vector is less than a preset velocity threshold;

[0336] If the ratio is greater than a preset ratio threshold, then the position change state is determined to be a motion state;

[0337] If the ratio is less than or equal to the preset ratio threshold, then the position change state is determined to be a stationary state.

[0338] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the first determining module is specifically used for:

[0339] Obtain the first camera pose of the first image and the second camera pose of the second image;

[0340] If the poses of the first camera and the poses of the second camera are inconsistent, then the position change state is determined to be a motion state.

[0341] If the pose of the first camera and the pose of the second camera are the same, then the position change state is determined to be a stationary state.

[0342] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the second determining module is specifically used for:

[0343] If the position change state is a motion state, then calculate the grid flow between the first image and the original reference image, and determine the grid flow as the candidate flow field;

[0344] If the position change state is a stationary state, then the zero-optical flow field is determined as the candidate flow field.

[0345] According to one or more embodiments of this disclosure, the image processing apparatus provided in this disclosure further includes:

[0346] The acquisition module is used to acquire multiple sets of image pairs with different grid resolutions based on the first image and the original reference image. Each set of image pairs includes a second image and an associated reference image. The second image is acquired by dividing the first image into grids at the corresponding grid resolution, and the associated reference image is acquired by scaling the original reference image according to the image size of the second image.

[0347] The update module is used to update the grid vertex positions of the corresponding second image according to the grid resolution in ascending order, when the current image pair has the smallest grid resolution, based on the corresponding second image and the associated reference image.

[0348] The update module is further configured to update the grid vertex position of the second image of the current image pair according to the grid vertex position of the previous image pair when the current image pair is not the minimum grid resolution, and update the grid vertex position of the corresponding second image according to the corresponding second image and the associated reference image.

[0349] The update module is also used to obtain the target mesh deformation image based on the mesh vertex position of the second image corresponding to the current image pair when the current image pair has the maximum mesh resolution;

[0350] The calculation module is used to calculate the mesh flow based on the target mesh deformation image and the first image.

[0351] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the acquisition module is specifically used for:

[0352] Based on the preset unit grid size and the multiple grid resolutions corresponding to the multiple sets of image pairs, the first image is divided into grids to generate multiple second images with different grid resolutions.

[0353] The original reference image is scaled according to the image size of the second image corresponding to each of the grid resolutions to generate a size-matched associated reference image;

[0354] The plurality of image pairs are generated based on the plurality of second images and the plurality of size-matched associated reference images.

[0355] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the updating module includes:

[0356] The calculation unit is used to iteratively calculate the residual value between the corresponding second image and the associated reference image based on the grid vertex position of the corresponding second image;

[0357] The judgment unit is used to determine whether the residual value of the current iteration meets the preset iteration stop condition during each iteration calculation.

[0358] The update unit is used to update the grid vertex position of the corresponding second image according to the residual value when the iteration stopping condition is not met, and continue iterative calculation until the residual value of the current iteration calculation meets the preset iteration stopping condition.

[0359] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the computing unit is specifically used for:

[0360] Based on the grid vertex positions of the corresponding second image, calculate the photometric error value between the corresponding second image and the associated reference image;

[0361] Calculate the regular expression value of the corresponding second image based on the grid vertex positions of the corresponding second image;

[0362] The residual value is determined based on the photometric error value and the regular expression value.

[0363] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the computing unit is specifically used for:

[0364] Determine multiple sample pixels of the corresponding second image;

[0365] Determine the grid where each sample pixel is located, and determine the pixel position of each sample pixel based on the grid vertex position of the grid where it is located;

[0366] The photometric error value is determined based on a preset photometric error calculation algorithm and the pixel position of the sample pixel.

[0367] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the computing unit is specifically used for:

[0368] The pixel gradient values ​​of each pixel in the second image at the current grid resolution are traversed one by one according to the preset step size;

[0369] Pixels whose pixel gradient values ​​are greater than or equal to the preset gradient threshold are identified as candidate sample pixels;

[0370] If the total number of candidate sample pixels is greater than a preset threshold, then the preset threshold number of candidate sample pixels are randomly selected as the sample pixels.

[0371] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, if the grid vertex positions of the corresponding second image have been updated, the calculation unit is specifically used for:

[0372] Each grid in the corresponding second image is divided into two triangles to obtain multiple triangles;

[0373] Based on the grid vertex positions of the corresponding second image, determine the first triangle vertex position of each triangle;

[0374] Obtain the position of the second vertex of each triangle in the corresponding second image before the most recent update;

[0375] The deformation error value of each triangle is determined based on the vertex positions of the first triangle and the vertex positions of the second triangle;

[0376] The regular expression value of the corresponding second image is determined based on the deformation error values ​​of all the triangles.

[0377] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the determining unit is specifically used for:

[0378] Determine whether the number of iterations in the current iteration is greater than or equal to the preset number of iterations for the current image pair, wherein the preset number of iterations is greater than 1;

[0379] If the number of iterations is greater than or equal to the preset number of iterations, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stopping condition.

[0380] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the determining unit is further configured to:

[0381] If the number of iterations is not greater than or equal to the preset number of iterations, then the historical residual value of the previous iteration corresponding to the current image pair is obtained;

[0382] Determine whether the residual value calculated in the current iteration is greater than or equal to the historical residual value;

[0383] If the residual value is greater than or equal to the historical residual value, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stop condition.

[0384] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the third computing module is specifically used for:

[0385] Based on the positions of the grid vertices in the target mesh deformation image, obtain the third pixel position of each pixel in the target mesh deformation image;

[0386] Based on the initial grid vertex positions of the second image corresponding to the maximum grid resolution, obtain the fourth pixel position of each pixel;

[0387] The grid flow is obtained by calculating the displacements of the third and fourth pixel positions of each pixel.

[0388] According to one or more embodiments of this disclosure, the image processing apparatus provided by this disclosure further includes: a downsampling module, used for:

[0389] Determine whether the resolution of the first image is greater than a preset resolution threshold;

[0390] If the resolution is greater than the preset resolution threshold, the first image is downsampled according to the preset downsampling ratio to obtain the downsampled first image.

[0391] The original reference image is downsampled according to the downsampling ratio to obtain the downsampled original reference image.

[0392] According to one or more embodiments of the present disclosure, the image processing apparatus provided by the present disclosure further includes: a downsampling module, configured to determine an upsampling ratio corresponding to the preset downsampling ratio;

[0393] The upsampled grid flow is obtained by upsampling the grid flow according to the upsampling ratio.

[0394] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the image alignment module is specifically used for:

[0395] The first image is aligned with the third image of the original reference image based on the optical flow.

[0396] Based on the candidate flow field, a fourth image is determined that aligns the first image with the original reference image;

[0397] Calculate the first photometric error image between the third image and the original reference image, and calculate the second photometric error image between the fourth image and the original reference image;

[0398] Based on the first photometric error image and the second photometric error image, a target flow is determined in the optical flow and the candidate flow field to facilitate alignment of the first image and the original reference image according to the target flow.

[0399] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the image alignment module is specifically used for:

[0400] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the image alignment module is specifically used for:

[0401] The original reference image is segmented into multiple superpixel blocks;

[0402] The first photometric error mean of the plurality of superpixel blocks is determined based on the first photometric error image;

[0403] The mean value of the second photometric error of the plurality of superpixel blocks is determined based on the second photometric error image;

[0404] Determine whether the mean value of the first photometric error is greater than the mean value of the second photometric error;

[0405] If the error is greater than the second mean photometric error, then the candidate flow field is determined to be the target flow.

[0406] If the optical flow is not greater than the second mean photometric error, then the optical flow is determined to be the target flow.

[0407] According to one or more embodiments of this disclosure, in the image processing apparatus provided by this disclosure, the image alignment module is specifically used for:

[0408] Determine the preset relaxation parameter value corresponding to the optical flow;

[0409] The preset relaxation parameter value and the mean value of the first photometric error are summed, and the mean value of the first photometric error is updated based on the summation result.

[0410] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0411] processor;

[0412] Memory used to store the processor's executable instructions;

[0413] The processor is configured to read the executable instructions from the memory and execute the instructions to implement any of the image processing methods provided in this disclosure.

[0414] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium storing a computer program for performing any of the image processing methods provided in the present disclosure.

[0415] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0416] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0417] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image processing method, characterized in that, Includes the following steps: Calculate the optical flow between the first image and the original reference image to be aligned; Determine the positional change state of the first image relative to the image background; If the position change state is a motion state, then the grid flow between the first image and the original reference image is calculated, and the grid flow is determined as a candidate flow field; If the position change state is a stationary state, then the zero-optical flow field is determined as the candidate flow field; Align the first image and the original reference image based on the optical flow and the candidate flow field.

2. The method as described in claim 1, characterized in that, The calculation of the optical flow between the first image and the original reference image to be aligned includes: Determine the first pixel position of each pixel in the first image, and determine the second pixel position of each pixel in the original reference image; Calculate the instantaneous velocity vector as the second pixel position moves to the first pixel position; The optical flow is obtained based on the instantaneous velocity vector of all pixels in the first image.

3. The method as described in claim 1, characterized in that, Determining the positional change state of the first image relative to the image background includes: The instantaneous velocity vector of each sample pixel in the first image is determined based on the optical flow; Determine the number of pixels whose magnitude of the instantaneous velocity vector is less than a preset velocity threshold; Calculate the ratio of the number of pixels to the total number of sample pixels in the first image; If the ratio is greater than a preset ratio threshold, then the position change state is determined to be a motion state; If the ratio is less than or equal to the preset ratio threshold, then the position change state is determined to be a stationary state.

4. The method as described in claim 1, characterized in that, Determining the positional change state of the first image relative to the image background includes: Obtain the first camera pose of the first image and the second camera pose of the second image; If the poses of the first camera and the poses of the second camera are inconsistent, then the position change state is determined to be a motion state. If the pose of the first camera and the pose of the second camera are the same, then the position change state is determined to be a stationary state.

5. The method as described in claim 1, characterized in that, The calculation of the grid flow between the first image and the original reference image includes: Multiple sets of image pairs with different grid resolutions are obtained based on the first image and the original reference image. Each image pair includes a second image and an associated reference image. The second image is obtained by dividing the first image into grids at a corresponding grid resolution. The associated reference image is obtained by scaling the original reference image according to the image size of the second image. According to the order of grid resolution from smallest to largest, when the current image pair has the smallest grid resolution, the grid vertex position of the corresponding second image is updated according to the corresponding second image and the associated reference image; When the current image pair is not the minimum grid resolution, update the grid vertex position of the second image corresponding to the current image pair according to the grid vertex position of the previous image pair, and update the grid vertex position of the corresponding second image according to the corresponding second image and the associated reference image; When the current image pair has the maximum grid resolution, the target grid deformation image is obtained based on the grid vertex position of the second image corresponding to the current image pair, and the grid flow is calculated based on the target grid deformation image and the first image.

6. The method as described in claim 5, characterized in that, The step of obtaining multiple sets of image pairs with different grid resolutions based on the first image and the original reference image includes: Based on the preset unit grid size and the multiple grid resolutions corresponding to the multiple sets of image pairs, the first image is divided into grids to generate multiple second images with different grid resolutions. The original reference image is scaled according to the image size of the second image corresponding to each of the grid resolutions to generate a size-matched associated reference image; The plurality of image pairs are generated based on the plurality of second images and the plurality of size-matched associated reference images.

7. The method as described in claim 5, characterized in that, The step of updating the mesh vertex positions of the corresponding second image based on the corresponding second image and the associated reference image includes: Based on the grid vertex positions of the corresponding second image, the residual values ​​between the corresponding second image and the associated reference image are calculated iteratively. During each iteration, determine whether the residual value of the current iteration meets the preset iteration stopping condition; If the iteration stopping condition is not met, the grid vertex position of the corresponding second image is updated according to the residual value, and the iteration calculation continues until the residual value of the current iteration calculation meets the preset iteration stopping condition.

8. The method as described in claim 7, characterized in that, The step of iteratively calculating the residual value between the corresponding second image and the associated reference image based on the grid vertex positions of the corresponding second image includes: Based on the grid vertex positions of the corresponding second image, calculate the photometric error value between the corresponding second image and the associated reference image; Calculate the regular expression value of the corresponding second image based on the grid vertex positions of the corresponding second image; The residual value is determined based on the photometric error value and the regular expression value.

9. The method as described in claim 8, characterized in that, The step of calculating the photometric error value between the corresponding second image and the associated reference image based on the grid vertex positions of the corresponding second image includes: Determine multiple sample pixels of the corresponding second image; Determine the grid where each sample pixel is located, and determine the pixel position of each sample pixel based on the grid vertex position of the grid where it is located; The photometric error value is determined based on a preset photometric error calculation algorithm and the pixel position of the sample pixel.

10. The method as described in claim 9, characterized in that, Determining the multiple sample pixels of the corresponding second image includes: The pixel gradient values ​​of each pixel in the second image at the current grid resolution are traversed one by one according to the preset step size; Pixels whose pixel gradient values ​​are greater than or equal to a preset gradient threshold are identified as candidate sample pixels; If the total number of candidate sample pixels is greater than a preset threshold, then the preset threshold number of candidate sample pixels are randomly selected as the sample pixels.

11. The method as described in claim 8, characterized in that, If the grid vertex positions of the corresponding second image have been updated, then calculating the regular expression value of the corresponding second image based on the grid vertex positions of the corresponding second image includes: Each grid in the corresponding second image is divided into two triangles to obtain multiple triangles; Based on the grid vertex positions of the corresponding second image, determine the first triangle vertex position of each triangle; Obtain the position of the second vertex of each triangle in the corresponding second image before the most recent update; The deformation error value of each triangle is determined based on the vertex positions of the first triangle and the vertex positions of the second triangle; The regular expression value of the corresponding second image is determined based on the deformation error values ​​of all the triangles.

12. The method as described in claim 7, characterized in that, The step of determining whether the residual value calculated in the current iteration meets the preset iteration stopping condition includes: Determine whether the number of iterations in the current iteration is greater than or equal to the preset number of iterations for the current image pair, wherein the preset number of iterations is greater than 1; If the number of iterations is greater than or equal to the preset number of iterations, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stopping condition.

13. The method as described in claim 12, characterized in that, Also includes: If the number of iterations is not greater than or equal to the preset number of iterations, then the historical residual value of the previous iteration corresponding to the current image pair is obtained; Determine whether the residual value calculated in the current iteration is greater than or equal to the historical residual value; If the residual value is greater than or equal to the historical residual value, then the residual value calculated in the current iteration is determined to satisfy the preset iteration stop condition.

14. The method as described in claim 5, characterized in that, The step of calculating the mesh flow based on the target mesh deformation image and the first image includes: Based on the positions of the grid vertices in the target mesh deformation image, obtain the third pixel position of each pixel in the target mesh deformation image; Based on the initial grid vertex positions of the second image corresponding to the maximum grid resolution, obtain the fourth pixel position of each pixel; The grid flow is obtained by calculating the displacements of the third and fourth pixel positions of each pixel.

15. The method as described in claim 6, characterized in that, Before generating multiple second images with different grid resolutions by dividing the first image into grids, the method further includes: Determine whether the resolution of the first image is greater than a preset resolution threshold; If the resolution is greater than the preset resolution threshold, the first image is downsampled according to the preset downsampling ratio to obtain the downsampled first image. The original reference image is downsampled according to the downsampling ratio to obtain the downsampled original reference image.

16. The method as described in claim 15, characterized in that, Before aligning the first image and the original reference image according to the optical flow and the mesh flow, the process includes: Determine the upsampling ratio corresponding to the preset downsampling ratio; The upsampled grid flow is obtained by upsampling the grid flow according to the upsampling ratio.

17. The method according to any one of claims 1-16, characterized in that, Aligning the first image and the original reference image based on the optical flow and the candidate flow field includes: The first image is aligned with the third image of the original reference image based on the optical flow. Based on the candidate flow field, a fourth image is determined that aligns the first image with the original reference image; Calculate the first photometric error image between the third image and the original reference image, and calculate the second photometric error image between the fourth image and the original reference image; Based on the first photometric error image and the second photometric error image, a target flow is determined in the optical flow and the candidate flow field to facilitate alignment of the first image and the original reference image according to the target flow.

18. The method as described in claim 17, characterized in that, The step of determining the target flow in the optical flow and the candidate flow field based on the first photometric error image and the second photometric error image includes: The original reference image is segmented into multiple superpixel blocks; The first photometric error mean of the plurality of superpixel blocks is determined based on the first photometric error image; The mean value of the second photometric error of the plurality of superpixel blocks is determined based on the second photometric error image; Determine whether the mean value of the first photometric error is greater than the mean value of the second photometric error; If the error is greater than the second mean photometric error, then the candidate flow field is determined to be the target flow. If the optical flow is not greater than the second mean photometric error, then the optical flow is determined to be the target flow.

19. The method as described in claim 18, characterized in that, Before determining whether the mean value of the first photometric error is greater than the mean value of the second photometric error, the method further includes: Determine the preset relaxation parameter value corresponding to the optical flow; The preset relaxation parameter value and the mean value of the first photometric error are summed, and the mean value of the first photometric error is updated based on the summation result.

20. An image processing apparatus, characterized in that, include: A calculation module is used to calculate the optical flow between the first image and the original reference image to be aligned; The first determining module is used to determine the positional change state of the first image relative to the image background; The second determining module is used to calculate the grid flow between the first image and the original reference image if the position change state is a motion state, and determine the grid flow as a candidate flow field; if the position change state is a stationary state, determine the zero-light flow field as the candidate flow field. An image alignment module is used to align the first image and the original reference image according to the optical flow and the candidate flow field.

21. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image processing method according to any one of claims 1-19.

22. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the image processing method according to any one of claims 1-19.

Images