Image registration methods, apparatus, storage media and electronic devices

Abstract

This application discloses an image registration method, apparatus, storage medium, and electronic device. The method includes: dividing a target frame image into multiple first image blocks according to a preset rule; calculating a motion model corresponding to each first image block; and registering the previous frame image of the target frame image based on each motion model. Using this application can improve the accuracy of image registration.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image registration method, apparatus, storage medium, and electronic device. Background Technology

[0002] Image registration is a common method in the field of image processing. In practice, due to the different distances between different objects and the camera device, the imaging positions of different objects will move different distances on the display interface of the camera device as the camera device moves. Summary of the Invention

[0003] This application provides an image registration method, apparatus, storage medium, and electronic device, which can improve the precision of image registration, i.e., improve the accuracy of image registration.

[0004] The technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide an image registration method, the method comprising:

[0006] The target frame image is divided into multiple first image blocks according to preset rules;

[0007] Calculate the motion model corresponding to each of the first image blocks;

[0008] The previous frame of the target frame image is registered based on each of the motion models.

[0009] Secondly, embodiments of this application provide an image registration apparatus, the apparatus comprising:

[0010] The image segmentation module is used to segment the target frame image into multiple first image blocks according to preset rules;

[0011] The model calculation module is used to calculate the motion model corresponding to each of the first image blocks;

[0012] The image registration module is used to register the previous frame image of the target frame image based on each of the motion models.

[0013] Thirdly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps of the first aspect described above.

[0014] Fourthly, embodiments of this application provide an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the method steps of the first aspect described above.

[0015] The beneficial effects of the technical solutions provided in some embodiments of this application include at least the following:

[0016] In this embodiment, the target frame image is divided into multiple first image blocks according to a preset rule, and the motion model corresponding to each first image block is calculated. Based on each motion model, the previous frame image of the target frame image is registered. Since each first image block corresponds to a motion model, the image registration of the previous frame image of the target frame image is performed based on the corresponding motion model. This avoids the error caused by perspective effect during the image registration of the whole image using a single motion model, and can improve the precision of image registration, that is, improve the accuracy of image registration. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an example schematic diagram of a perspective effect provided in an embodiment of this application;

[0019] Figure 2 This is a schematic flowchart of an image registration method provided in an embodiment of this application;

[0020] Figure 3 This is an example schematic diagram of image segmentation provided in an embodiment of this application;

[0021] Figure 4 This is an example schematic diagram of a target frame image provided in an embodiment of this application;

[0022] Figure 5 This is an example diagram illustrating feature point selection provided in an embodiment of this application;

[0023] Figure 6 This is an example schematic diagram of image segmentation provided in an embodiment of this application;

[0024] Figure 7 This is an example schematic diagram of image registration provided in an embodiment of this application;

[0025] Figure 8 This is a schematic flowchart of an image registration method provided in an embodiment of this application;

[0026] Figure 9 This is an example schematic diagram illustrating the effect of a motion model provided in an embodiment of this application;

[0027] Figure 10 This is a schematic flowchart of an image registration method provided in an embodiment of this application;

[0028] Figure 11 This is a schematic diagram of the structure of an image registration device provided in an embodiment of this application;

[0029] Figure 12 This is a schematic diagram of the structure of a model calculation module provided in an embodiment of this application;

[0030] Figure 13 This is a schematic diagram of the structure of a model calculation unit provided in an embodiment of this application;

[0031] Figure 14 This is a schematic diagram of the structure of a model calculation unit provided in an embodiment of this application;

[0032] Figure 15 This is a schematic diagram of the structure of an image registration device provided in an embodiment of this application;

[0033] Figure 16 This is a schematic diagram of the structure of an image registration module provided in an embodiment of this application;

[0034] Figure 17 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0037] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0038] When a camera device using a mobile terminal or a camera device communicating with a mobile terminal is used to photograph an object, the handheld nature of the mobile terminal may cause shaking or movement, resulting in a change in the image position of the object on the corresponding display screen of the camera device. Figure 1 As shown in the figure, the black dots represent the photographed object, and the distance X that the image of the photographed object moves across the display screen represents its movement. L -X R It can be calculated using the following formula:

[0039]

[0040] Therefore, we can know the distance X that the camera device has been moved. L -X R Furthermore, when the distance v between the camera and the image device is fixed, the distance the image of the object moves on the display screen is related to the object distance, i.e., the distance between the camera and the object. When the object distance u is large, the distance X the image of the object moves on the display screen is also large. L -X R When the object distance u is small, the distance X that the image of the photographed object moves on the display interface is relatively small. L -X R The error is relatively large. Due to the existence of the aforementioned perspective effect, using a single motion model to register images will result in errors because different objects move different distances on the display interface.

[0041] Based on this, this application provides an image registration method that divides an image into multiple image blocks, calculates the motion model corresponding to each of the multiple image blocks, and uses the motion model to register the previous frame image corresponding to the image, which can improve the accuracy of image registration.

[0042] This method can be implemented using a computer program and can run on an image registration device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application. The image registration device in this embodiment can be a mobile terminal, including but not limited to: smart interactive flat panels, personal computers, tablets, handheld devices, in-vehicle devices, wearable devices, computing devices, or other processing devices connected to a wireless modem. In different networks, user terminals can have different names, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user equipment, cellular phone, cordless phone, personal digital assistant (PDA), terminal device in 5G networks or future evolved networks, etc.

[0043] The present application will now be described in detail with reference to specific embodiments.

[0044] The implementation scenario of the image registration method is as follows: a user uses the camera of a mobile terminal to capture an image of an object, obtaining the current frame image. The mobile terminal can acquire the image captured by the camera and use the current frame image as the reference frame image in the image registration process. The reference frame image is segmented according to a preset segmentation rule to obtain multiple segmented image blocks. Based on the reference frame image and the segmented image blocks of the previous frame image of the reference frame image, the corresponding motion model for image registration is calculated. Based on its motion model, the previous frame image of the reference frame image is image registered to obtain the previous frame image of the reference frame image. This registered image is input to the user interface of the mobile terminal so that the user can view the image after image registration. If the previous frame image of the reference frame image acquired by the camera has no motion, the calculated motion model is a no-motion model, and image registration of the previous frame image of the reference frame image is not required. The model can be directly output to the user interface of the mobile terminal for the user to view.

[0045] Please see Figure 2 This is a schematic flowchart illustrating an image registration method provided in an embodiment of this application. The embodiments of this application are described from the perspective of an image registration device, and the image registration method may include the following steps:

[0046] S101, the target frame image is divided into multiple first image blocks according to preset rules;

[0047] The preset rule is a pre-defined image segmentation method, which can be segmented by row, by column, or a combination of rows and columns. For example, the image can be segmented into 4×1 image blocks, 4×4 image blocks, or 3×3 image blocks, etc. The number of image blocks and their specific shapes are not limited here.

[0048] The target frame image can be the current frame image captured by the camera device of the mobile terminal.

[0049] After the target frame image is divided into multiple image blocks according to the preset rules, each of the divided image blocks is the first image block.

[0050] It is also understandable that if the hardware processing method of the image is row scanning, then a row of the image is a first image block; if the hardware processing method of the image is column scanning, then a column of the image is a first image block.

[0051] S102, calculate the motion model corresponding to each of the first image blocks respectively;

[0052] Each first image block corresponds to a motion model. The process of calculating the motion model corresponding to each first image block is to perform feature point detection and feature point matching on the feature points of the first image block, and then calculate the motion model of each first image block based on the results of feature point detection and feature point matching.

[0053] Obtain multiple second image blocks from the previous frame of the target frame image according to the preset rules; based on the multiple first image blocks and the second image blocks corresponding to each first image block, calculate the motion model corresponding to each first image block.

[0054] The second image block is the result of segmenting the previous frame of the target frame image according to a preset rule. The segmentation rule for the first image block is the same as that for the second image block. For example, if the previous frame of the target frame image is segmented into 4×4 second image blocks according to the preset segmentation rule, then the target frame image is also segmented into 4×4 first image blocks according to the preset segmentation rule. Figure 3 The result shown is the segmentation, which yields 16 first image blocks.

[0055] In one embodiment, multiple first image blocks are traversed sequentially to obtain a first feature point set corresponding to the currently traversed target first image block, the first feature point set containing all first feature points; a target second image block matching the target first image block is determined among multiple second image blocks, and a second feature point set matching the first feature point set is determined in the target second image block, each first feature point in the first feature point set matching each second feature point in the second feature point set; a motion model corresponding to the target first image block is calculated based on the first feature point set and the second feature point set; until all first image blocks have been traversed.

[0056] Here, the target first image patch is the first image patch that is currently traversed and will be used for image registration. All feature points in the target first image patch are called first feature points, and there may be multiple first feature points.

[0057] The second image patch that matches the target first image patch refers to the second image patch among multiple second image patches that corresponds to the coordinate position of the target first image patch, for example, such as... Figure 4 As shown, the image on the right is the target frame image, and the image on the left is the previous frame image of the target frame image. When the coordinates of the first target image block in the target frame image are (1, 1), then the image block that matches the first target image block is searched in the second image block in the previous frame image of the target frame image. The search result can be the second target image block corresponding to the coordinates (1', 1') in the second image block.

[0058] Obtain the set of first feature points corresponding to the first image block of the target currently being traversed. The set of first feature points includes all first feature points. The feature values ​​of the pixels in the image can be calculated using the Harris angle method. The Harris angle is calculated by subtracting the determinant value of matrix M from the trace of M and then comparing the difference with a pre-given threshold. Alternatively, the Shi-Tomasi algorithm, the Smallest Univalue Segment Assimilating Nucleus (SUSAN), or other methods can be used to calculate the feature values ​​of the pixels in the image.

[0059] After calculating the feature values ​​of the pixels in the target first image block, a first feature point set corresponding to the target first image block is obtained and the feature value corresponding to each first feature point in the first feature point set is calculated; first feature points whose feature values ​​are less than a preset feature threshold are filtered out.

[0060] The preset feature threshold can be a pre-set empirical feature value, which can be used to filter out feature points with low feature values.

[0061] The method for filtering out the first feature points whose feature values ​​are less than the preset feature threshold is to compare the calculated feature value with the preset feature threshold. If the feature value is less than the preset feature threshold, it needs to be filtered out, leaving the first feature points whose feature values ​​are greater than the preset feature threshold.

[0062] In this scheme, one first image block corresponds to one motion model. However, when calculating the motion model of a first image block, there may be a situation where the number of feature points of the target first image block is insufficient. Furthermore, this scheme can use an affine transformation model, which requires at least three non-collinear feature points for calculation. Therefore, after filtering out the first feature points whose feature values ​​are less than a preset feature threshold, the number of feature points whose feature values ​​are greater than the preset feature threshold is obtained. It is then determined whether the number of feature points greater than the preset feature threshold is greater than a preset number. If the number of feature points greater than the preset feature threshold is greater than the preset number, the motion model can be calculated based on the first feature points after filtering out the first feature points whose feature values ​​are less than the preset feature threshold.

[0063] If the number is less than a preset number, then the feature points whose feature values ​​are greater than the preset feature value, and the feature points in the first image blocks surrounding the target first image block whose feature values ​​are greater than the preset feature value, are taken as the first feature points. For example, such as Figure 5As shown, since the motion models of the first image blocks on the left, top, upper left, and upper right sides of the target first image block (1,1) have been obtained according to the traversal order from the first row to the last row, and the image to the right of the target first image block is the image block needed to calculate the image block in this row, the feature points corresponding to the feature values ​​of the first image blocks on the left, right, top, upper left, and upper right sides of the target first image block with feature values ​​greater than the preset feature values ​​are used. This ensures that the speed of calculating the model of the target first image block is not greatly affected or is not affected, thus ensuring that this scheme can improve the image registration speed.

[0064] Before calculating the motion model of the target first image block, first sub-image blocks are created with each filtered first feature point as the center and a preset length as the side length, thus dividing the target frame image into segments as follows: Figure 6 The multiple first sub-image blocks shown can be created in the same way for the second sub-image blocks. Second sub-image blocks are created with each second feature point as the center and the preset length as the side length. Motion vectors from the target first sub-image block to the target second sub-image block are calculated based on each first sub-image block and each second sub-image block. Motion models corresponding to the target first image block are calculated based on each motion vector and using the least squares method.

[0065] Since the second image patch matches the first image patch, each first feature point in the first image patch corresponds one-to-one with each second feature point in the second image patch. The number of first feature points is the same as the number of second feature points. The coordinates of the first feature point in the first image patch can be the same as or similar to the coordinates of the second feature point in the second image patch. For example, if the coordinates of a feature point in the first sub-image patch are (10.1, 1.5), there is a matching feature point in the second sub-image patch with coordinates (10.1, 1.6). The feature values ​​of the two matching feature points must be the same or similar.

[0066] In one embodiment, when traversing to the target first sub-image block in the first sub-image block, the image block in the second sub-image block that is most similar to the content in the target first sub-image block is searched and selected as the target second sub-image block. This is the process of feature point matching. The similarity between the target first sub-image block and the second sub-image block can be calculated by using methods such as the Mean Absolute Differences (MAD) algorithm or the Structural Similarity Index Measure (SSIM), which are not limited here.

[0067] Based on each of the first and second sub-image blocks, the motion vector from the target second sub-image block to the target first sub-image block is calculated. For example, when the coordinates of the target first sub-image block are (x, y) and the coordinates of the target second sub-image block are (x', y'), the motion vector corresponding to the target first sub-image block can be calculated using the formula mv. x =x'-x,mv y =y'-y, that is, the motion vector corresponding to the first sub-image patch of the target is (mv x ,mv y ).

[0068] S103, register the previous frame image of the target frame image based on each of the motion models.

[0069] like Figure 7 As shown, the second image blocks corresponding to multiple first image blocks are registered based on the motion model corresponding to each first image block in the target frame image. The registration method can be to input the coordinates of feature points in the target first sub-image block, and calculate the target coordinates of the corresponding feature points in the target second sub-image block in the previous frame image based on the model corresponding to the target first image block where the target first sub-image block is located, and assign the pixel value of the target coordinates to the feature points corresponding to the target second sub-image block.

[0070] In this embodiment, the target frame image is divided into multiple first image blocks according to preset rules. A motion model corresponding to each first image block is calculated. Based on each motion model, the previous frame image of the target frame image is registered. This allows for direct calculation of the motion model corresponding to the first image block after feature point detection and matching, without waiting for feature point detection and matching of all features in the target frame image. Subsequently, image registration is performed on the previous frame image based on the calculated motion model, improving the efficiency of image registration. Since each first image block corresponds to a motion model, image registration based on the corresponding motion model avoids errors caused by perspective effects during image registration of the entire image using a single motion model, thus improving the precision and accuracy of image registration.

[0071] Please see Figure 8 This is a schematic flowchart of an image registration method provided in an embodiment of this application. In this embodiment, after calculating the motion model corresponding to the first image block, the motion model is verified to further determine its reliability. This embodiment is described from the perspective of an image registration device, and the image registration method may include the following steps:

[0072] S201, the target frame image is divided into multiple first image blocks according to preset rules;

[0073] Please refer to S101; it will not be repeated here.

[0074] S202, calculate the motion model corresponding to each of the first image blocks respectively;

[0075] Please refer to S102; it will not be repeated here.

[0076] S203, Verify the effectiveness of the motion model and obtain the verification results;

[0077] The calculated motion model needs further reliability verification. When verifying the target first image block currently being traversed, the corresponding four-corner offsets in the motion model of the target first image block can be verified. If the four-corner offsets are less than a preset four-corner offset, the verification result indicates that the motion model is a valid motion model; otherwise, it is an invalid motion model. Alternatively, the corresponding four-sided deflection degree in the motion model of the target first image block can be verified. If the four-sided deflection degree is less than a preset four-sided deflection degree, the verification result indicates that the motion model is a valid motion model; otherwise, it is an invalid motion model.

[0078] like Figure 9 As shown, rectangle ABCD is obtained by deforming the target first image block using the calculated motion model, resulting in rectangle A'B'C'D'.

[0079] like Figure 9 As shown, the four corner offsets are the sum of the lengths of the four line segments AA', BB', CC', and DD'. If the four corner offsets are less than the preset four corner offsets, the verification result indicates that the motion model is a valid motion model; otherwise, it is an invalid motion model.

[0080] The four-sided deflection degree is the sum of the angles of the four angles θ1, θ2, θ3, and θ4 in the figure. When the four-sided deflection degree is less than the preset four-sided deflection degree, the verification result indicates that the motion model is a valid motion model; otherwise, it is an invalid motion model.

[0081] S204, if the verification result indicates that the motion model is a valid motion model, then the previous frame image of the target frame image is registered based on each of the motion models; if the verification result indicates that the motion model is an invalid motion model, then the motion model is corrected, and the previous frame image of the target frame image is registered based on the corrected motion model.

[0082] If the verification result indicates that the motion model is an invalid motion model, the motion model can be corrected by selecting the motion model of an adjacent image block of the target first image block as the motion model of the target first image block, and the motion model of the adjacent image block is an effective motion model; or, the motion model corresponding to the target first image block is defined as a no-motion model, and the previous frame image of the target frame image is not registered.

[0083] For example, the motion model of the image block to the left or right of the target first image block can be used as the motion model of the target first image block. However, the motion model of the selected adjacent image block must be a valid motion model. When a valid motion model cannot be found even if the motion model of an adjacent image block of the target first image block is used as the motion model of the target first image block, the motion model of the target first image block is defined as ((1,0,0), (0,1,0)), that is, no motion model. This means that the target first image block has not moved and there is no need to register the target second image block corresponding to the target first image block.

[0084] In this embodiment, the target frame image is divided into multiple first image blocks according to preset rules. A motion model is calculated for each first image block, and the validity of the motion model is verified to obtain a verification result. If the verification result indicates that the motion model is valid, the previous frame image of the target frame image is registered based on each motion model. If the verification result indicates that the motion model is invalid, the motion model is corrected, and the previous frame image of the target frame image is registered based on the corrected motion model. After obtaining the motion model corresponding to the first image block, the motion model is verified to further determine its reliability and avoid using invalid motion models for image registration, thus improving the accuracy of image registration. After performing feature point detection and feature point matching steps on the first image block, the calculation of the motion model corresponding to the first image block can proceed directly without waiting for all feature point detection and feature point matching of the target frame image. Subsequently, image registration is performed on the previous frame image of the target image based on the calculated motion model, improving the efficiency of image registration. Since each first image block corresponds to a motion model, image registration is performed on the previous frame of the target frame image based on the corresponding motion model. This avoids errors caused by perspective effects during the image registration of the entire image using a single motion model, thereby improving the precision of image registration, i.e., improving the accuracy of image registration.

[0085] Please see Figure 10 This is a flowchart illustrating an image registration method provided in an embodiment of this application.

[0086] In this embodiment, an affine transformation model is employed, and the target coordinates of the feature points corresponding to the previous frame image of the target frame image are calculated according to a preset formula. The previous frame image of the target frame image is then registered based on these target coordinates. This embodiment is described from the perspective of an image registration device, and the image registration method may include the following steps:

[0087] S301, the target frame image is divided into multiple first image blocks according to preset rules;

[0088] Please refer to S101; it will not be repeated here.

[0089] S302, calculate the motion model corresponding to each of the first image blocks respectively;

[0090] Please refer to S102; it will not be repeated here.

[0091] S303, calculate the target coordinates of the feature points corresponding to the previous frame image of the target frame image based on the coordinates of the feature points in the target frame image and the motion model; replace the pixel value corresponding to the reference coordinates of the feature points in the previous frame image of the target frame image with the pixel value of the target coordinates.

[0092] The motion model used in this scheme is the affine transformation model. An affine transformation is a linear transformation between two-dimensional coordinates, preserving the "flatness" of the two-dimensional graphic. According to the calculation formula corresponding to the affine transformation model:

[0093]

[0094] It can calculate the target coordinates of the feature points corresponding to the previous frame of the target frame image.

[0095] in, The matrix is ​​an affine matrix, where X and Y are the coordinates of the feature points in the target frame image, and X′ and Y′ are the target coordinates of the feature points corresponding to the previous frame image.

[0096] This application employs a method where a target frame image is segmented into multiple first image blocks according to preset rules. A motion model is calculated for each first image block. Based on the coordinates of feature points in the target frame image and the motion model, the target coordinates of the feature points corresponding to the previous frame image are calculated. The coordinates of the feature points in the previous frame image are then replaced with the target coordinates. This process is repeated for all feature points in the target frame image to complete the registration of the previous frame image. After feature point detection and matching for each first image block, the calculation of the motion model corresponding to that first image block can proceed directly, without waiting for feature point detection and matching of all feature points in the target frame image. Subsequently, image registration of the previous frame image is performed based on the calculated motion model, improving the efficiency of image registration. Because each first image block corresponds to a motion model, image registration of the previous frame image based on the corresponding motion model avoids errors caused by perspective effects during image registration of the entire image using a single motion model, thus improving the precision and accuracy of image registration.

[0097] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0098] Please see Figure 11 This illustration shows a schematic diagram of an image registration device provided in an exemplary embodiment of this application. The image registration device can be implemented as all or part of a mobile terminal through software, hardware, or a combination of both. The image registration device 1 includes an image segmentation module 11, a model calculation module 12, and an image registration module 13, wherein:

[0099] Image segmentation module 11 is used to segment the target frame image into multiple first image blocks according to preset rules;

[0100] Model calculation module 12 is used to calculate the motion model corresponding to each of the first image blocks respectively;

[0101] The image registration module 13 is used to register the previous frame image of the target frame image based on each of the motion models.

[0102] Optional, please see Figure 12 The model calculation module 12 includes:

[0103] The second image block acquisition unit 121 is used to acquire multiple second image blocks of the previous frame image of the target frame image according to the preset rules;

[0104] The model calculation unit 122 is used to calculate the motion model corresponding to each first image block based on the plurality of first image blocks and the second image block corresponding to each first image block.

[0105] Optional, please see Figure 13 The model calculation unit 122 includes:

[0106] The feature point acquisition subunit 1221 is used to traverse multiple first image blocks sequentially and acquire the first feature point set corresponding to the target first image block currently being traversed.

[0107] The feature point determination subunit 1222 is used to determine a target second image block that matches the target first image block among a plurality of second image blocks, and to determine a second feature point set that matches the first feature point set in the target second image block, wherein each first feature point in the first feature point set matches each second feature point in the second feature point set.

[0108] The model calculation subunit 1223 is used to calculate the motion model corresponding to the target first image block based on the first feature point set and the second feature point set;

[0109] Complete traversal of subunit 1224, used until all first image blocks have been traversed.

[0110] Optionally, the model calculation subunit 1223 is specifically used for:

[0111] With each of the first feature points as the center and a preset length as the side length, a first sub-image block is created respectively, and a second sub-image block is created with each of the second feature points as the center and the preset length as the side length.

[0112] Calculate the motion vector from the target first sub-image block to the target second sub-image block based on each of the first sub-image blocks and each of the second sub-image blocks;

[0113] Based on the motion vectors and using the least squares method, the motion model corresponding to the first image block of the target is calculated.

[0114] Optionally, the feature point acquisition subunit 1221 is specifically used for:

[0115] Obtain the first feature point set corresponding to the target first image block and calculate the feature value corresponding to each first feature point in the first feature point set;

[0116] The first feature point whose feature value is less than a preset feature threshold is filtered out.

[0117] Optional, please see Figure 14The model calculation unit 122 further includes:

[0118] The quantity acquisition subunit 1225 is used to acquire the number of feature points whose feature values ​​are greater than a preset feature threshold.

[0119] The first feature point determination subunit 1226 is used to, if the number is less than a preset number, select feature points whose feature values ​​are greater than the preset feature value, and feature points in the first image block surrounding the target first image block whose feature values ​​are greater than the preset feature value as first feature points.

[0120] Optional, please see Figure 15 The device 1 further includes:

[0121] The model verification unit 14 is used to verify the effectiveness of the motion model and obtain the verification result;

[0122] The image registration module 13 includes:

[0123] Image registration unit 131 is used to register the previous frame image of the target frame image based on each of the motion models if the verification result indicates that the motion model is a valid motion model.

[0124] The image registration unit 131 is further configured to, if the verification result indicates that the motion model is an invalid motion model, correct the motion model and register the previous frame image of the target frame image based on the corrected motion model.

[0125] Optionally, the model verification unit 14 includes:

[0126] The model verification subunit 141 is used to verify the corresponding four-corner offsets in the motion model of the target first image block. If the four-corner offsets are less than the preset four-corner offsets, the verification result indicates that the motion model is a valid motion model; otherwise, the verification result indicates that the motion model is an invalid motion model.

[0127] The model verification subunit 141 is also used to verify the quadrilateral deflection degree corresponding to the motion model of the target first image block. If the quadrilateral deflection degree is less than the preset quadrilateral deflection degree, the verification result indicates that the motion model is a valid motion model; otherwise, the verification result indicates that the motion model is an invalid motion model.

[0128] Optionally, the image registration unit 131 further includes:

[0129] Image registration subunit 1311 is configured to, if the verification result indicates that the motion model is an invalid motion model, replace the motion model of the target first image patch with the motion model of an adjacent image patch of the target first image patch, and register the previous frame image of the target frame image based on the motion model of the adjacent image patch, wherein the motion model of the adjacent image patch is a valid motion model, or...

[0130] The image registration subunit 1311 is further configured to define the target first image block as a motionless model and not register the previous frame image of the target frame image if the verification result indicates that the motion model is an invalid motion model and the motion model of the target first image block cannot be replaced by the motion model of an adjacent image block of the target first image block.

[0131] Optional, please see Figure 16 The image registration module 13 includes:

[0132] The target coordinate calculation unit 131 is used to calculate the target coordinates of the feature points corresponding to the previous frame image of the target frame image based on the coordinates of the feature points in the target frame image and the motion model.

[0133] The coordinate replacement unit 132 is used to replace the pixel value corresponding to the reference coordinate of the feature point in the previous frame image of the target frame image with the pixel value of the target coordinate.

[0134] Optionally, the target coordinate calculation unit 131 includes:

[0135] Target coordinate calculation subunit 1311 is used to apply the formula

[0136]

[0137] Calculate the target coordinates of the feature points corresponding to the previous frame of the target frame image.

[0138] in, The matrix is ​​an affine matrix, where X and Y are the coordinates of the feature points in the target frame image, and X′ and Y′ are the target coordinates of the feature points corresponding to the previous frame image.

[0139] In this embodiment, the target frame image is divided into multiple first image blocks according to preset rules. A motion model is calculated for each first image block, and the validity of the motion model is verified to obtain a verification result. If the verification result indicates that the motion model is valid, the previous frame image of the target frame image is registered based on each motion model. If the verification result indicates that the motion model is invalid, the motion model is corrected, and the previous frame image of the target frame image is registered based on the corrected motion model. After obtaining the motion model corresponding to the first image block, the motion model is verified to further determine its reliability and avoid using invalid motion models for image registration, thus improving the accuracy of image registration. After performing feature point detection and feature point matching steps on the first image block, the calculation of the motion model corresponding to the first image block can proceed directly without waiting for the detection and matching of all feature points in the target frame image. Subsequently, image registration is performed on the previous frame image of the target frame image based on the calculated motion model, improving the efficiency of image registration. Since each first image block corresponds to a motion model, image registration is performed on the previous frame of the target frame image based on the corresponding motion model. This avoids errors caused by perspective effects during the image registration of the entire image using a single motion model, thereby improving the precision of image registration, i.e., improving the accuracy of image registration.

[0140] It should be noted that the image registration device provided in the above embodiments is only illustrated by the division of the above functional modules when performing the image registration method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image registration device and the image registration method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0141] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0142] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0143] This application also provides an electronic device, in which the robot stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-10 The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figures 1-10 The specific details of the illustrated embodiments will not be elaborated here.

[0144] Please see Figure 17 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 17 As shown, the electronic device 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0145] The communication bus 1002 is used to realize the connection and communication between these components.

[0146] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0147] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0148] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 1001 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 1001.

[0149] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 17 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a robot positioning application.

[0150] exist Figure 17In the mobile terminal 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and to obtain the user's input data; while the processor 1001 can be used to call the image registration application stored in the memory 1005 and specifically perform the following operations:

[0151] The target frame image is divided into multiple first image blocks according to preset rules;

[0152] Calculate the motion model corresponding to each of the first image blocks;

[0153] The previous frame of the target frame image is registered based on each of the motion models.

[0154] In one embodiment, when the processor 1001 performs the calculation of the motion model corresponding to each of the first image blocks, it specifically performs the following operations:

[0155] Obtain multiple second image blocks from the previous frame of the target frame image, segmented according to the preset rules;

[0156] Based on the plurality of first image blocks and the second image blocks corresponding to each first image block, the motion model corresponding to each first image block is calculated respectively.

[0157] In one embodiment, when the processor 1001 calculates the motion model corresponding to each first image block based on the plurality of first image blocks and the second image block corresponding to each first image block, it specifically performs the following operations:

[0158] Iterate through multiple first image blocks in sequence to obtain the set of first feature points corresponding to the target first image block currently being traversed;

[0159] Among a plurality of second image blocks, a target second image block matching the target first image block is determined, and a second feature point set matching the first feature point set is determined in the target second image block, wherein each first feature point in the first feature point set is respectively matched with each second feature point in the second feature point set;

[0160] Calculate the motion model corresponding to the first image block of the target based on the first set of feature points and the second set of feature points;

[0161] This continues until all first image blocks have been traversed.

[0162] In one embodiment, when the processor 1001 performs the operation of calculating the motion model corresponding to the target first image patch based on the first feature point set and the second feature point set, the processor 1001 specifically performs the following operations:

[0163] With each of the first feature points as the center and a preset length as the side length, a first sub-image block is created respectively, and a second sub-image block is created with each of the second feature points as the center and the preset length as the side length.

[0164] Calculate the motion vector from the target first sub-image block to the target second sub-image block based on each of the first sub-image blocks and each of the second sub-image blocks;

[0165] Based on the motion vectors and using the least squares method, the motion model corresponding to the first image block of the target is calculated.

[0166] In one embodiment, the processor 1001 performs the following operations when executing the process of obtaining the first feature point set corresponding to the currently traversed target first image block:

[0167] Obtain the first feature point set corresponding to the target first image block and calculate the feature value corresponding to each first feature point in the first feature point set;

[0168] The first feature point whose feature value is less than a preset feature threshold is filtered out.

[0169] In one embodiment, after the processor 1001 performs the operation of obtaining the first feature point set corresponding to the currently traversed target first image block, it further performs the following operations:

[0170] Obtain the number of feature points whose feature values ​​are greater than a preset feature threshold;

[0171] If the quantity is less than the preset quantity, then the feature points whose feature values ​​are greater than the preset feature values, and the feature points in the first image blocks surrounding the target first image block whose feature values ​​are greater than the preset feature values, are taken as the first feature points.

[0172] In one embodiment, after the processor 1001 performs the calculation of the motion model corresponding to each of the first image blocks, it further performs the following operations:

[0173] The effectiveness of the motion model was verified, and the verification results were obtained.

[0174] Registration of the previous frame image of the target frame image based on each of the aforementioned motion models includes:

[0175] If the verification result indicates that the motion model is a valid motion model, then the previous frame image of the target frame image is registered based on each of the motion models;

[0176] If the verification result indicates that the motion model is an invalid motion model, then the motion model is corrected, and the previous frame of the target frame image is registered based on the corrected motion model.

[0177] In one embodiment, when the processor 1001 performs the verification of the validity of the motion model and obtains the verification result, it specifically performs the following operations:

[0178] Verify the corner offsets corresponding to the motion model of the first image block of the target. If the corner offsets are less than the preset corner offsets, the verification result indicates that the motion model is a valid motion model; otherwise, the verification result indicates that the motion model is an invalid motion model.

[0179] The quadrilateral deflection degree corresponding to the motion model of the first image block of the target is verified. If the quadrilateral deflection degree is less than the preset quadrilateral deflection degree, the verification result indicates that the motion model is a valid motion model; otherwise, the verification result indicates that the motion model is an invalid motion model.

[0180] In one embodiment, when the processor 1001 performs the following operations: if the verification result indicates that the motion model is an invalid motion model, then correct the motion model and register the previous frame image of the target frame image based on the corrected motion model:

[0181] If the verification result indicates that the motion model is invalid, then the motion model of the target first image patch is replaced with the motion model of a neighboring image patch. Based on the motion model of the neighboring image patch, the previous frame of the target frame image is registered. The motion model of the neighboring image patch is then considered a valid motion model.

[0182] If the verification result indicates that the motion model is an invalid motion model, and the motion model of the target first image block cannot be replaced with the motion model of an adjacent image block of the target first image block, then the target first image block is defined as a no-motion model, and the previous frame image of the target frame image is not registered.

[0183] In one embodiment, when the processor 1001 performs registration of the previous frame image of the target frame image based on each of the motion models, it specifically performs the following operations:

[0184] The target coordinates of the feature points in the target frame image are calculated based on the coordinates of the feature points in the previous frame image of the target frame image and the motion model.

[0185] Replace the pixel value corresponding to the reference coordinates of the feature point in the previous frame of the target frame image with the pixel value of the target coordinates.

[0186] In one embodiment, the motion model is an affine transformation model. When the processor 1001 executes the calculation of the target coordinates of the feature points corresponding to the previous frame image based on the coordinates of the feature points in the target frame image and the motion model, it specifically performs the following operations:

[0187] The target coordinates of the feature points corresponding to the previous frame image of the target frame image are calculated using a preset formula;

[0188] Wherein, the preset formula is Let X and Y be the coordinates of the feature points in the target frame image, and X′ and Y′ be the target coordinates of the feature points corresponding to the previous frame image.

[0189] In this embodiment, the target frame image is divided into multiple first image blocks according to preset rules. A motion model is calculated for each first image block, and the validity of the motion model is verified to obtain a verification result. If the verification result indicates that the motion model is valid, the previous frame image of the target frame image is registered based on each motion model. If the verification result indicates that the motion model is invalid, the motion model is corrected, and the previous frame image of the target frame image is registered based on the corrected motion model. After obtaining the motion model corresponding to the first image block, the motion model is verified to further determine its reliability and avoid using invalid motion models for image registration, thus improving the accuracy of image registration. After performing feature point detection and feature point matching steps on the first image block, the calculation of the motion model corresponding to the first image block can proceed directly without waiting for all feature point detection and feature point matching of the target frame image. Subsequently, image registration is performed on the previous frame image of the target image based on the calculated motion model, improving the efficiency of image registration. Since each first image block corresponds to a motion model, image registration of the previous frame of the target image based on the corresponding motion model avoids the error caused by perspective effect during the image registration of the whole image using a single motion model, which can improve the precision of image registration, that is, improve the accuracy of image registration.

[0190] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0191] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An image registration method, characterized in that, The method includes: The target frame image is divided into multiple first image blocks according to preset rules; Calculate the motion model corresponding to each of the first image blocks; The effectiveness of the motion model was verified, and the verification results were obtained. If the verification result indicates that the motion model is a valid motion model, then the previous frame image of the target frame image is registered based on each of the motion models; the motion model is used to calculate the coordinates of the corresponding feature points in the previous frame image of the target frame image based on the coordinates of the feature points in the target frame image; during the registration process of the previous frame image of the target frame image, the corresponding second image block is registered based on the motion model corresponding to each first image block in the target frame image; If the verification result indicates that the motion model is an invalid motion model, then the motion model is corrected, and the previous frame image of the target frame image is registered based on the corrected motion model; The verification of the effectiveness of the motion model, and the obtaining of the verification results, includes: Verify the four corner offsets corresponding to the motion model of the first image block of the target. If the four corner offsets are less than the preset four corner offsets, the verification result indicates that the motion model is a valid motion model; otherwise, the verification result indicates that the motion model is an invalid motion model; or; Verify the quadrilateral deflection degree corresponding to the motion model of the first image block of the target. If the quadrilateral deflection degree is less than the preset quadrilateral deflection degree, the verification result indicates that the motion model is a valid motion model; otherwise, the verification result indicates that the motion model is an invalid motion model. If the verification result indicates that the motion model is an invalid motion model, then the motion model is corrected, and the previous frame image of the target frame image is registered based on the corrected motion model, including: If the verification result indicates that the motion model is invalid, then the motion model of the target first image patch is replaced with the motion model of a neighboring image patch. Based on the motion model of the neighboring image patch, the previous frame of the target frame image is registered. The motion model of the neighboring image patch is then considered a valid motion model. If the verification result indicates that the motion model is an invalid motion model, and the motion model of the target first image block cannot be replaced with the motion model of an adjacent image block of the target first image block, then the target first image block is defined as a no-motion model, and the previous frame image of the target frame image is not registered.

2. The method according to claim 1, characterized in that, The step of calculating the motion model corresponding to each of the first image blocks includes: Obtain multiple second image blocks from the previous frame of the target frame image, segmented according to the preset rules; Based on the plurality of first image blocks and the second image blocks corresponding to each first image block, the motion model corresponding to each first image block is calculated respectively.

3. The method according to claim 2, characterized in that, The step of calculating the motion model corresponding to each of the plurality of first image blocks and the second image blocks corresponding to each of the first image blocks includes: Iterate through multiple first image blocks in sequence to obtain the set of first feature points corresponding to the target first image block currently being traversed; Among a plurality of second image blocks, a target second image block matching the target first image block is determined, and a second feature point set matching the first feature point set is determined in the target second image block, wherein each first feature point in the first feature point set is respectively matched with each second feature point in the second feature point set; Calculate the motion model corresponding to the first image block of the target based on the first set of feature points and the second set of feature points; This continues until all first image blocks have been traversed.

4. The method according to claim 3, characterized in that, The step of calculating the motion model corresponding to the target first image patch based on the first feature point set and the second feature point set includes: With each of the first feature points as the center and a preset length as the side length, a first sub-image block is created respectively, and a second sub-image block is created with each of the second feature points as the center and the preset length as the side length. Calculate the motion vector from the target first sub-image block to the target second sub-image block based on each of the first sub-image blocks and each of the second sub-image blocks; Based on the motion vectors and using the least squares method, the motion model corresponding to the first image block of the target is calculated.

5. The method according to claim 3, characterized in that, The step of obtaining the set of first feature points corresponding to the currently traversed target first image patch includes: Obtain the first feature point set corresponding to the target first image block and calculate the feature value corresponding to each first feature point in the first feature point set; The first feature point whose feature value is less than a preset feature threshold is filtered out.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the number of feature points whose feature values ​​are greater than a preset feature threshold; If the number is less than a preset number, then the feature points whose feature values ​​are greater than the preset feature threshold, and the feature points whose feature values ​​are greater than the preset feature threshold in the first image block surrounding the target first image block, are taken as the first feature points.

7. The method according to claim 1, characterized in that, The registration of the previous frame image of the target frame image based on each of the motion models includes: The target coordinates of the feature points in the target frame image are calculated based on the coordinates of the feature points in the previous frame image of the target frame image and the motion model. Replace the pixel value corresponding to the reference coordinates of the feature point in the previous frame of the target frame image with the pixel value of the target coordinates.

8. The method according to claim 7, characterized in that, The motion model is an affine transformation model. The calculation of the target coordinates of the feature points corresponding to the feature points in the previous frame image based on the coordinates of the feature points in the target frame image and the motion model includes: The target coordinates of the feature points corresponding to the previous frame image of the target frame image are calculated using a preset formula; The preset formula is [ , [ ] is an affine matrix, and X and Y are the coordinates of feature points in the target frame image. , The target coordinates are the feature points corresponding to the previous frame of the target frame image.

9. An image registration device, characterized in that, The device includes: The image segmentation module is used to segment the target frame image into multiple first image blocks according to preset rules; The model calculation module is used to calculate the motion model corresponding to each of the first image blocks; the model verification unit is used to verify the effectiveness of the motion model and obtain the verification result. The image registration module includes an image registration unit, which is configured to: if the verification result indicates that the motion model is a valid motion model, register the previous frame image of the target frame image based on each of the motion models; if the verification result indicates that the motion model is an invalid motion model, correct the motion model and register the previous frame image of the target frame image based on the corrected motion model; and during the registration process of the previous frame image of the target frame image, register the corresponding second image block based on the motion model corresponding to each first image block in the target frame image. The model calculation module further includes a model verification subunit, which is used to verify the four-corner offsets corresponding to the motion model of the first image block of the target frame image. If the four-corner offsets are less than a preset four-corner offset, the verification result indicates that the motion model is a valid motion model; otherwise, the verification result indicates that the motion model is an invalid motion model. The model verification subunit is used to verify the quadrilateral deflection degree corresponding to the motion model of the target first image block. If the quadrilateral deflection degree is less than the preset quadrilateral deflection degree, the verification result indicates that the motion model is a valid motion model; otherwise, the verification result indicates that the motion model is an invalid motion model. The image registration unit includes an image registration subunit, which is used to: if the verification result indicates that the motion model is an invalid motion model, replace the motion model of the target first image block with the motion model of an adjacent image block of the target first image block, and register the previous frame image of the target frame image based on the motion model of the adjacent image block, wherein the motion model of the adjacent image block is a valid motion model, or... The image registration subunit is further configured to define the target first image block as a no-motion model and not register the previous frame image of the target frame image if the verification result indicates that the motion model is an invalid motion model and the motion model of the target first image block cannot be replaced by the motion model of an adjacent image block of the target first image block.

10. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1-8.

11. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the method steps as claimed in any one of claims 1-8.

Images