Image registration method, device, equipment and storage medium

Abstract

The disclosed embodiments relate to an image registration method, apparatus, device and storage medium, the method comprising: obtaining an image to be registered of a target area and first three-dimensional coordinates of a plurality of marker points set in the target area, wherein the first three-dimensional coordinates are determined based on the image to be registered; generating first feature descriptors of the plurality of marker points based on the image to be registered and the first three-dimensional coordinates; aligning the image to be registered with a reference image of the target area based on the first feature descriptor and second feature descriptors of the plurality of marker points, wherein the second feature descriptor is generated based on the reference image and second three-dimensional coordinates of the plurality of marker points, and the second three-dimensional coordinates are determined based on the reference image. The image registration method provided by the disclosed embodiments is suitable for image registration scenarios with low color and texture.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image technology, and in particular to an image registration method, apparatus, device and storage medium. Background Art

[0002] Image registration refers to the process of aligning and matching two or more images obtained at different times, from different sensors, and / or under different conditions (such as weather, lighting, shooting position and angle, etc.). Image registration can be used to merge or fuse two or more images for more in-depth analysis. It is widely used in medical, industrial and other technical fields.

[0003] The current image registration process is to first extract features from the image, and then register two or more images based on the extracted features. However, this method has high requirements on the color and texture of the image. When the image lacks color and texture changes, feature extraction becomes very difficult, making it difficult to achieve registration. In other words, this method is not suitable for low-color and low-texture image registration scenarios. For example, in the surgical robot navigation system, real-time capture of the position and motion information of the human body is a very important link, especially for the case where the camera and actuator are fixed at the end of the robotic arm (i.e., the eye is on the hand). As the end of the robotic arm approaches the human body, the camera itself cannot capture the overall image of the human body, but can only capture partial local images, such as partial skin images of the back, face, and buttocks of the human body. However, the skin area is often smooth and lacks color and texture changes. Therefore, it becomes very difficult to extract features and register it. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide an image registration method, apparatus, device and storage medium.

[0005] A first aspect of an embodiment of the present disclosure provides an image registration method, the method comprising:

[0006] Acquire the image to be registered of the target area and first three-dimensional coordinates of a plurality of marking points set in the target area, wherein the first three-dimensional coordinates are determined based on the image to be registered;

[0007] Generate first feature descriptors of a plurality of marking points based on the image to be registered and the first three-dimensional coordinates;

[0008] Based on the first feature descriptor and the second feature descriptor of the multiple marking points, the image to be registered and the reference image of the target area are aligned, wherein the second feature descriptor is generated based on the reference image and the second three-dimensional coordinates of the multiple marking points, and the second three-dimensional coordinates are determined based on the reference image.

[0009] A second aspect of an embodiment of the present disclosure provides an image registration device, the device comprising:

[0010] An acquisition module, used for acquiring the image to be registered of the target area and first three-dimensional coordinates of a plurality of marking points set in the target area, wherein the first three-dimensional coordinates are determined based on the image to be registered;

[0011] A generating module, used for generating first feature descriptors of a plurality of marking points based on the image to be registered and the first three-dimensional coordinates;

[0012] An alignment module is used to align the image to be registered and the reference image of the target area based on a first feature descriptor and a second feature descriptor of multiple marking points, wherein the second feature descriptor is generated based on the reference image and the second three-dimensional coordinates of the multiple marking points, and the second three-dimensional coordinates are determined based on the reference image.

[0013] A third aspect of an embodiment of the present disclosure provides an electronic device, the server comprising: a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method of the first aspect above.

[0014] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of the first aspect described above can be implemented.

[0015] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0016] The disclosed embodiments can obtain an image to be registered of a target area and first three-dimensional coordinates of a plurality of marking points set in the target area, wherein the first three-dimensional coordinates are determined based on the image to be registered; generate first feature descriptors of the plurality of marking points based on the image to be registered and the first three-dimensional coordinates; align the image to be registered and a reference image of the target area based on the first feature descriptor and second feature descriptors of the plurality of marking points, wherein the second feature descriptor is generated based on the reference image and second three-dimensional coordinates of the plurality of marking points, and the second three-dimensional coordinates are determined based on the reference image. It can be seen that by adopting the above technical scheme, multiple marking points can be artificially set in the target area, so that both the image to be registered and the reference image include the marking points. In this way, the features of the target area can be encoded based on the first three-dimensional coordinates of the image to be registered and the multiple marking points to generate a first feature descriptor. Similarly, the features of the target area can be encoded based on the reference image and the second three-dimensional coordinates of the multiple marking points to generate a second feature descriptor, and then the image to be registered and the reference image can be aligned based on the first feature descriptor and the second feature descriptor. In this way, even if the target area lacks color and texture changes, the features of the target area can be encoded and registered based on the multiple marking points artificially set in the target area. Therefore, the image registration method provided in the embodiment of the present disclosure is suitable for registration scenarios of low color and texture images. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 is a flow chart of an image registration method provided by an embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram of a sector-shaped partition provided by an embodiment of the present disclosure;

[0021] Figure 3 is a logical schematic diagram of an image registration process provided by an embodiment of the present disclosure;

[0022] Figure 4 is a structural schematic diagram of an image registration device provided by an embodiment of the present disclosure;

[0023] Figure 5 It is a structural diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0025] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0026] Figure 1 is a flow chart of an image registration method provided by an embodiment of the present disclosure, and the method can be performed by an electronic device. The electronic device can be exemplarily understood as a device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, etc. Figure 1 As shown, the method provided in this embodiment includes the following steps:

[0027] S110, acquiring an image to be registered of a target area and first three-dimensional coordinates of a plurality of marking points set in the target area, wherein the first three-dimensional coordinates are determined based on the image to be registered.

[0028] Specifically, the target area is the area that needs to be encoded and registered, which may include any area lacking color and texture changes, and illustratively, the target area may be a human skin area, a ground remote sensing area, an industrial product surface, a building wall, etc., but is not limited thereto.

[0029] Specifically, the image to be registered is obtained by an image acquisition device that acquires an image of a target area. The image acquisition device includes, for example, a camera (such as an RGBD stereo camera), a camera head, etc., which is not limited thereto. In addition, the image acquisition device can be integrated into an electronic device or can be independent of the electronic device, which is not limited thereto.

[0030] Specifically, the specific total number, shape and distribution of the marking points can be set by those skilled in the art according to actual conditions and are not limited here. For example, n (n is greater than or equal to 2) circular marking points are marked randomly or according to scene requirements in the target area, but it is not limited thereto.

[0031] Specifically, the first three-dimensional coordinates are the three-dimensional coordinates of the marking point in the coordinate system of the image acquisition device.

[0032] Specifically, those skilled in the art may determine the first three-dimensional coordinates based on the image to be registered in any possible manner, which is not limited to this.

[0033] For example, n markers are set in the target area, and an RGBD stereo camera is used to capture an image of the target area (the image to be matched or the reference image described below). Then, the pixel coordinates of the n markers are detected using the OpenCV-based spot detection algorithm. Then, the camera coordinate conversion SDK is called to obtain the three-dimensional coordinates of the n markers (the first three-dimensional coordinates or the second three-dimensional coordinates described below), which are recorded as (P1, P2…P n ), where P i is the three-dimensional coordinate of the i-th marker point.

[0034] S120: Generate first feature descriptors of a plurality of marking points based on the image to be registered and the first three-dimensional coordinates.

[0035] Specifically, a feature descriptor is a mathematical vector used to represent the characteristics of a target area.

[0036] It should be noted that the specific implementation logic of “generating a first feature descriptor of multiple marking points based on the image to be registered and the first three-dimensional coordinates” and “generating a second feature descriptor of multiple marking points based on the reference image and the second three-dimensional coordinates” are the same. Several typical examples of this implementation logic (i.e., “generating feature descriptors of multiple marking points based on the image and three-dimensional coordinates”) are described in detail below, but do not constitute a limitation of the present disclosure.

[0037] In some embodiments, feature descriptors of multiple marker points are generated based on an image and three-dimensional coordinates, including: inputting the image and the three-dimensional coordinates into a trained first neural network model, and obtaining feature descriptors of the multiple marker points output by the first neural network model, wherein when the image is an image to be registered, the feature descriptor is a first feature descriptor, and when the image is a reference image, the feature descriptor is a second feature descriptor.

[0038] In some other embodiments, based on the image and the three-dimensional coordinates, generating feature descriptors of multiple marker points includes: taking the multiple marker points one by one as the current marker point, and generating the feature descriptor of the current marker point through the following steps, wherein when the image is the image to be registered, the feature descriptor is the first feature descriptor, and when the image is the reference image, the feature descriptor is the second feature descriptor:

[0039] S121. Determine the main direction of the current marking point based on the image.

[0040] Specifically, the main direction is a direction calculated around the current marker point in the image. After the main direction is determined, the features around the current marker point can be described as information relative to the main direction, so that the final feature descriptor remains unchanged when the image is rotated, that is, the robustness and consistency of the feature descriptor are improved.

[0041] The main direction is the direction of the line connecting the current mark point and the target mark point. Exemplarily, the main direction is the direction from the current mark point to the target mark point in the image, or the main direction is the direction from the target mark point to the current mark point in the image, which is not limited to this.

[0042] The target marking point is one of the surrounding marking points of the current marking point, and the surrounding marking points are marking points other than the current marking point among the multiple marking points.

[0043] Exemplarily, n marking points are set in the target area. If the current marking point is the i-th marking point, then the 1st to (i-1)th marking points and the (i+1)th to nth marking points are surrounding marking points.

[0044] In some examples, the target marker point is the surrounding marker point with the smallest Euclidean distance to the current marker point among all surrounding marker points.

[0045] Specifically, the Euclidean distance between two marking points is, in the coordinate system of the image acquisition device, the straight-line distance between the two marking points.

[0046] It can be understood that among the surrounding marking points around the current marking point, the closer the surrounding marking points are to the current marking point, the lower the degree of distortion relative to the current marking point. Therefore, by setting the target marking point as the surrounding marking point with the smallest Euclidean distance to the current marking point among all surrounding marking points, the calculated main direction can be made closer to the actual situation, thereby making the subsequently calculated feature descriptors more accurate, which is beneficial to improving the accuracy of image registration.

[0047] Of course, in other examples, the target marking point may be a surrounding marking point having the largest Euclidean distance to the current marking point among all surrounding marking points.

[0048] S122. For each surrounding marker point except the target marker point, determine a preset clockwise angle between the secondary direction and the primary direction of the surrounding marker point based on the image, and use it as the angle label value of the surrounding marker point relative to the primary direction.

[0049] The secondary direction of a surrounding marker point is the direction corresponding to the line connecting the current marker point and the surrounding marker point. Exemplarily, the secondary direction is the direction from the current marker point to the surrounding marker point in the image, or the secondary direction is the direction from the surrounding marker point to the current marker point in the image, which is not limited to this.

[0050] Specifically, the preset clockwise direction may be a clockwise direction or a counterclockwise direction, and there is no limitation on this.

[0051] Exemplarily, there are n marking points set in the target area. If the current marking point is the i-th marking point and the target marking point is the k-th marking point, then for each surrounding marking point except the "k-th marking point" in the "1st to (i-1)th marking points and (i+1)th to nth marking points", the preset clockwise angle between the secondary direction and the main direction of each surrounding marking point is determined, so as to obtain the angle label value of each surrounding marking point relative to the main direction, that is, the angle label value of the (i-2) surrounding marking points is obtained.

[0052] S123. Determine an initial feature descriptor of the current marking point based on the angle tag value.

[0053] Specifically, the initial feature descriptor is a feature descriptor preliminarily calculated based on “the angle label value of each surrounding marking point except the target marking point”.

[0054] In some examples, S123 includes: for each equally divided sector in the circle with the current marking point as the center, determining all angle label values ​​within the angle interval corresponding to the equally divided sector and adding them up to obtain the elements corresponding to the equally divided sector; based on the elements, determining the initial feature descriptor.

[0055] Specifically, the circle with the current mark point as the center can be equally divided into a plurality of sectors (i.e., equally divided sectors), i.e., the sector angles of the sectors are the same. The specific total number of equally divided sectors can be set by those skilled in the art according to actual conditions and is not limited here.

[0056] Specifically, the equally divided sector is composed of an arc and two radii connecting the two ends of the arc. The angle interval corresponding to an equally divided sector includes two endpoint values, one of which is a preset clockwise angle between one of the radii in the equally divided sector and the main direction, and the other is a preset clockwise angle between another radius in the equally divided sector and the main direction.

[0057] Specifically, the total number of equally divided sectors is the same as the vector dimension of the feature descriptor. For an equally divided sector, the sum of the angle label values ​​of all surrounding marking points within the equally divided sector is the element corresponding to the equally divided sector.

[0058] Specifically, the elements corresponding to the multiple equally divided sectors are arranged according to the preset rules, and a multidimensional vector (i.e., a feature descriptor) can be obtained. The specific content of the preset rules can be set by those skilled in the art according to actual conditions, and are not limited here. Exemplarily, the preset rule can be that the smaller the endpoint value of the angle interval corresponding to the equally divided sector is, the smaller the subscript of the element corresponding to the equally divided sector is; or; the preset rule can be that the larger the endpoint value of the angle interval corresponding to the equally divided sector is, the smaller the subscript of the element corresponding to the equally divided sector is, but it is not limited thereto.

[0059] For example, Figure 2 Schematic diagram of a sector partition provided by an embodiment of the present disclosure. Figure 2 , the current mark point is the i-th mark point, the k-th mark point is the mark point with the smallest Euclidean distance to the i-th mark point, and in the image, the direction from the i-th mark point to the k-th mark point is the main direction. The circle with the i-th mark point as the center is divided into 20 sectors (i.e., equally divided sectors), and the sector angle of each equally divided sector is 18°. The angle interval of the first equally divided sector is [0, 18°), the angle interval of the second equally divided sector is [18°, 36°), the angle interval of the third equally divided sector is [36°, 54°), and so on. The angle interval of the twentieth equally divided sector is [342°, 360°). For the first equally divided sector, all angle label values ​​in the angle interval corresponding to the first equally divided sector are summed to obtain the element corresponding to the first equally divided sector; for the second equally divided sector, all angle label values ​​in the angle interval corresponding to the second equally divided sector are summed to obtain the element corresponding to the second equally divided sector; for the third equally divided sector, all angle label values ​​in the angle interval corresponding to the third equally divided sector are summed to obtain the element corresponding to the third equally divided sector, and so on, to obtain 20 elements. The 20 elements are arranged according to preset rules to obtain the initial feature descriptor of the kth marker point.

[0060] It can be understood that by setting the main direction, the angle label value of each surrounding marking point except the target marking point is calculated based on the main direction, and then for each equally divided sector, the sum of the angle label values ​​of all surrounding marking points located in the equally divided sector is used as the element corresponding to the equally divided sector, so as to determine the initial feature descriptor based on the elements corresponding to multiple equally divided sectors, so that the calculation method of the initial feature descriptor is simple and convenient, and the initial feature descriptor can be made rotationally invariant, that is, the robustness and consistency of the initial feature descriptor are improved.

[0061] Of course, in other examples, S123 may include: inputting the angle label value of each surrounding marker point except the target marker point into a pre-trained second neural network model, and obtaining the initial feature descriptor of the current marker point output by the second neural network model.

[0062] S124: Based on the three-dimensional coordinates, the initial feature descriptor is scaled to obtain a feature descriptor of the current marking point.

[0063] Specifically, by performing scale correction on the initial feature descriptor, the feature descriptor of the current marker point can be made scale invariant.

[0064] In some examples, S124 includes: S1241. Based on the three-dimensional coordinates, obtain the Euclidean distance from the current marker point to each surrounding marker point; S1242. Add the Euclidean distances to obtain the total Euclidean distances, and select the minimum value from the Euclidean distances to obtain the minimum Euclidean distance; S1243. For each surrounding marker point except the target marker point, determine the eigenvalue weight of the surrounding marker point relative to the current marker point based on the total Euclidean distance, the minimum Euclidean distance, the preset scale coefficient and the total number of multiple marker points, wherein the larger the Euclidean distance between the surrounding marker point and the current marker point, the larger the corresponding eigenvalue weight; S1244. For each eigenvalue weight, add it with the corresponding element to correct the initial feature description scale to obtain the feature descriptor of the current marker point.

[0065] Specifically, for each surrounding marker point, the Euclidean distance from the current marker point to the surrounding marker points is calculated. All calculated Euclidean distances are summed to obtain the total Euclidean distance. The minimum value is selected from all Euclidean distances to obtain the minimum Euclidean distance.

[0066] Exemplarily, there are n marking points set in the target area. If the current marking point is the i-th marking point, the Euclidean distance from the current marking point to the surrounding marking points is calculated for each surrounding marking point in "the 1st to (i-1)th marking points and the (i+1)th to nth marking points", and (n-1) Euclidean distances are obtained. The (n-1) Euclidean distances are added to obtain the total Euclidean distance, and the minimum value is selected from the (n-1) Euclidean distances to obtain the minimum Euclidean distance.

[0067] Specifically, the eigenvalue weight reflects the degree of variation of the angle label values ​​corresponding to the surrounding marker points due to scale changes. In other words, the eigenvalue weight makes the feature descriptor scale-invariant, which can cope with the situation where the distance between the image acquisition device and the registration object changes.

[0068] Optionally, for each surrounding marking point except the target marking point, based on the sum of the Euclidean distances, the minimum Euclidean distance, the preset scale factor and the total number of the plurality of marking points, determining the feature value weight of the surrounding marking point relative to the current marking point, including: for each surrounding marking point except the target marking point, calculating the feature value weight of the surrounding marking point relative to the current marking point using the following formula:

[0069]

[0070] Among them, w is the feature value weight, n is the total number of multiple marking points, D sum is the sum of the Euclidean distances, D min is the minimum Euclidean distance, β is the preset scale factor, and D is the Euclidean distance from the current marker point to the surrounding marker points.

[0071] Specifically, the specific value of β can be tested and adjusted according to the actual application scenario and is not limited here.

[0072] Specifically, for each surrounding marker point except the target marker point, the feature value weight of the surrounding marker point relative to the current marker point is added to the element corresponding to the “equally divided sector corresponding to the angle label value of the surrounding marker point”.

[0073] Exemplarily, there are n marking points set in the target area. If the current marking point is the i-th marking point, the feature value weight of the j-th (j≠k) marking point relative to the current marking point is as follows:

[0074]

[0075] Among them, w i j is the feature value weight of the jth (j≠k) marking point relative to the current marking point, n is the total number of marking points, D i sum is the sum of the Euclidean distances corresponding to the i-th marked point (the current marked point), D i min is the minimum Euclidean distance corresponding to the i-th marker point, β is the preset scale factor, D ij is the Euclidean distance from the ith marker point to the jth marker point. The feature value weight w of the ith marker point (surrounding marker points) relative to the ith marker point ij Add it to the element corresponding to the equally divided sector corresponding to the angle label value of the j-th marking point.

[0076] Of course, in other examples, S1244 can also be replaced by S1245, and for each surrounding marker point except the target marker point, its angle label value and its corresponding feature value weight are added to update the angle label value, and the feature descriptor of the current marker point is determined based on the updated angle label value.

[0077] It can be understood that the closer the distance between the surrounding marker points and the current marker point, the smaller the deviation between the corresponding angle label value and the actual situation is, and the smaller the possibility of being lost (that is, the smaller the possibility of not being seen by the image acquisition device is); the farther the distance between the surrounding marker points and the current marker point, the greater the deviation between the corresponding angle label value and the actual situation is, and it may even be lost (that is, the image acquisition device may not be able to see it). Therefore, by setting the closer the distance between the surrounding marker points and the current marker point, the larger the eigenvalue weight is, the closer the distance between the surrounding marker points and the current marker point, the greater the impact on the feature descriptor of the current marker point, and the farther the distance between the surrounding marker points and the current marker point, the smaller the impact on the feature descriptor of the current marker point. In this way, even if the surrounding marker points that are far away from the current marker point are lost due to scale transformation, the impact on the feature descriptor is relatively small and can be ignored, so that the feature descriptor of the current marker point has scale invariance.

[0078] Of course, in some other examples, S124 includes: inputting the three-dimensional coordinates and initial feature descriptors of multiple marking points into a pre-trained third neural network model, and obtaining the feature descriptors of the current marking points output by the third neural network model.

[0079] S130. Align the image to be registered with a reference image of the target area based on the first feature descriptor and the second feature descriptor of the multiple marking points, wherein the second feature descriptor is generated based on the reference image and second three-dimensional coordinates of the multiple marking points, and the second three-dimensional coordinates are determined based on the reference image.

[0080] Specifically, the image to be registered and the reference image of the target area are aligned, that is, the two are unified into the same coordinate system.

[0081] In some embodiments, S130 includes: determining a plurality of matching landmark point pairs of the image and the reference image based on the first feature descriptor and the second feature descriptor; and determining a transformation matrix for aligning the image and the reference image based on the plurality of matching landmark point pairs.

[0082] Specifically, those skilled in the art may obtain multiple matching landmark point pairs in any possible manner. For example, the feature descriptor matching algorithm BruteForcrMatcher() integrated in OpenCV is used to match the feature descriptors in the reference image with those in the image to be registered, thereby completing the point-to-point registration process.

[0083] Specifically, those skilled in the art can obtain the transformation matrix in any possible way. For example, according to the corresponding three-dimensional coordinates of multiple matching landmark points and based on the least squares method, the rigid transformation matrix can be solved to complete the registration of the reference image and the image to be registered.

[0084] The image registration method provided by the present disclosure is described in detail below based on a specific example. Figure 3 is a logical diagram of an image registration process provided by an embodiment of the present disclosure, see Figure 3 , the image registration process includes the following steps:

[0085] 1. Mark n points in the target area that needs to be encoded and registered. You can mark them manually and randomly, or mark them according to scene requirements.

[0086] 2. Use an RGBD stereo camera to capture images of the target area (reference frame), use the blob detection algorithm based on OpenCV to detect the pixel coordinates of n markers, and then call the camera coordinate conversion SDK to obtain the three-dimensional coordinates of the n markers.

[0087] 3. Calculate the feature descriptor of each marker point, which is a 20-dimensional vector V.

[0088] Here, the process of calculating the feature descriptor is described by taking the i-th marker point as an example. The calculation process of the feature descriptor Vi of the i-th marker point is as follows:

[0089] 3.1. Calculate the Euclidean distance from all other marked points (i.e. surrounding marked points) to the i-th marked point, and record the sum of the Euclidean distances from the i-th marked point to all other marked points as The minimum Euclidean distance is The minimum Euclidean distance is the Euclidean distance to the kth marked point.

[0090] The direction from the i-th marker point to the k-th marker point is set as the main direction. For each marker point except the k-th marker point, the counterclockwise angle between the direction from the i-th marker point to the marker point (referred to as the secondary direction) and the main direction is used as the angle label value corresponding to the marker point. With the i-th marker point as the center, the image area is divided into 20 sectors, with each 18° angle sector as one area (such as Figure 2 As shown in Figure 1.1 ). The marking point is divided into 20 sectors according to its angle label value. For each sector, the angle label value corresponding to it is summed to obtain its corresponding element. The 20 elements corresponding to the 20 sectors are arranged into the initial feature descriptor Vi0 of the i-th marking point. This operation is to make the feature descriptor rotationally invariant and improve the robustness of the algorithm.

[0091] 3.2. Calculate the eigenvalue weights of other marking points relative to the i-th marking point. Taking the j-th marking point as an example, the eigenvalue weight calculation formula is:

[0092]

[0093] in:

[0094]

[0095]

[0096] β is the preset scale factor, which is tested and adjusted according to the actual application scenario. ij is the feature value weight of the jth (j≠k≠i) marker relative to the current marker, D ij is the Euclidean distance from the i-th marker point to the j-th marker point. The eigenvalue weight representation has the characteristic of scale invariance and can cope with the situation where the distance between the camera and the registration object changes.

[0097] 3.3. Update the initial feature descriptor Vi0. The feature value weight w of the jth marker relative to the i-th marker ij Add it to the mth element of the vector Vi0 (the angle label of the jth marker point relative to the ith marker point is added to obtain the mth element). The same is true for the feature value weights of other marker points, and no further details are given until the final feature descriptor vector Vi of the ith marker point is updated.

[0098] Repeat steps 3.1-3.3 until the feature descriptor representation matrix M of all marked points is calculated, and its size is n rows and 20 columns.

[0099] 4. According to the actual application scenario, the target area image is collected again during the movement of the robot arm, and the feature descriptor representation matrix of the marked points in the image (frame to be registered) is calculated again.

[0100] 5. Use the descriptor matching algorithm BruteForcrMatcher() integrated in OpenCV to match the feature descriptors in the reference image with those in the image to be registered to complete the marker-to-marker registration process.

[0101] 6. According to the corresponding coordinates of the front and back matching mark points, the rigid transformation matrix is ​​solved based on the least squares method to complete the registration of the front and back target areas.

[0102] The disclosed embodiment proposes a feature descriptor generation algorithm, which can successfully encode and align the features of the block area by only marking a limited number of marking points. Traditional algorithms require that there are relatively obvious color and texture differences in the image near the feature points for encoding and specificity. In this scene, the color and texture color of each image are relatively close and there is no obvious difference, so the scene cannot be processed. The present invention can specifically encode the marking points and perform high-precision matching based on the positional relationship between a few feature points. Compared with traditional common algorithms, the present invention can not only process this kind of low-color and low-texture images, but also has the characteristics of scale invariance, rotation invariance, etc., so it has higher robustness and stability.

[0103] Figure 4 is a structural diagram of an image registration device provided by an embodiment of the present disclosure, and the image registration device can be understood as the above-mentioned electronic device or a part of the functional modules in the above-mentioned electronic device. Figure 4 As shown, the image registration device 400 includes:

[0104] An acquisition module 410 is used to acquire an image to be registered of a target area and first three-dimensional coordinates of a plurality of marking points set in the target area, wherein the first three-dimensional coordinates are determined based on the image to be registered;

[0105] A generating module 420, configured to generate first feature descriptors of the plurality of marking points based on the image to be registered and the first three-dimensional coordinates;

[0106] The alignment module 430 is used to align the image to be registered and the reference image of the target area based on the first feature descriptor and the second feature descriptors of the multiple marking points, wherein the second feature descriptor is generated based on the reference image and the second three-dimensional coordinates of the multiple marking points, and the second three-dimensional coordinates are determined based on the reference image.

[0107] Optionally, the generating module 420 is used to generate feature descriptors of the plurality of marking points based on the image and the three-dimensional coordinates, wherein the generating module 420 includes a first determining submodule, a second determining submodule, a third determining submodule and a fourth determining submodule;

[0108] The multiple marking points are taken as current marking points one by one, and the feature descriptors of the current marking points are generated through the first determination submodule, the second determination submodule, the third determination submodule and the correction submodule:

[0109] A first determination submodule is configured to determine a main direction of the current marker point based on the image, wherein the main direction is a direction corresponding to a line connecting the current marker point and a target marker point, the target marker point is one of the surrounding marker points of the current marker point, and the surrounding marker points are marker points other than the current marker point among the multiple marker points;

[0110] A second determination submodule is used to determine, for each of the surrounding marking points except the target marking point, a preset clockwise angle between the secondary direction of the surrounding marking point and the main direction based on the image, and use it as an angle label value of the surrounding marking point relative to the main direction, wherein the secondary direction is a direction corresponding to a line connecting the current marking point and the surrounding marking points;

[0111] A third determination submodule, configured to determine an initial feature descriptor of the current marking point based on the angle tag value;

[0112] A correction submodule, configured to perform scale correction on the initial feature descriptor based on the three-dimensional coordinates to obtain the feature descriptor of the current marking point;

[0113] Wherein, when the image is the image to be registered, the feature descriptor is a first feature descriptor, and when the image is the reference image, the feature descriptor is a second feature descriptor.

[0114] Optionally, the target marking point is a surrounding marking point having the smallest Euclidean distance to the current marking point among all the surrounding marking points.

[0115] Optionally, the third determination submodule is specifically used to determine, for each equally divided sector in the circle with the current marking point as the center, all angle label values ​​located in the angle interval corresponding to the equally divided sector and add them up to obtain the element corresponding to the equally divided sector;

[0116] Based on the elements, the initial feature descriptor is determined.

[0117] Optionally, the correction submodule includes:

[0118] A first acquisition unit, configured to acquire a Euclidean distance from the current marking point to each of the surrounding marking points based on the three-dimensional coordinates;

[0119] A second acquisition unit is used to add the Euclidean distances to obtain a total Euclidean distance, and select a minimum value from the Euclidean distances to obtain a minimum Euclidean distance;

[0120] a determining unit, configured to determine, for each of the surrounding marking points except the target marking point, a feature value weight of the surrounding marking point relative to the current marking point based on the sum of the Euclidean distances, the minimum Euclidean distance, a preset scale factor, and the total number of the plurality of marking points, wherein the larger the Euclidean distance between the surrounding marking point and the current marking point, the larger the corresponding feature value weight;

[0121] A correction unit is used to add each of the feature value weights to the corresponding element to correct the initial feature description scale to obtain the feature descriptor of the current marking point.

[0122] Optionally, the determining unit is specifically configured to calculate, for each of the surrounding marking points except the target marking point, the feature value weight of the surrounding marking point relative to the current marking point using the following formula:

[0123]

[0124] Wherein, w is the feature value weight, n is the total number of the multiple marking points, D sum is the sum of the Euclidean distances, D min is the minimum Euclidean distance, β is the preset scale factor, and D is the Euclidean distance from the current marking point to the surrounding marking points.

[0125] Optionally, the alignment module 430 is specifically configured to determine a plurality of matching landmark point pairs of the image and the reference image based on the feature descriptor and the second feature descriptor;

[0126] Based on the plurality of matching landmark point pairs, a transformation matrix for aligning the image with the reference image is determined.

[0127] The device provided in this embodiment can execute the method of any of the above embodiments, and its execution method and beneficial effects are similar, which will not be repeated here.

[0128] An embodiment of the present disclosure further provides an electronic device, which includes: a memory, in which a computer program is stored; and a processor, for executing the computer program. When the computer program is executed by the processor, the method of any of the above embodiments can be implemented.

[0129] For example, Figure 5 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure. The electronic device 500 in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0130] like Figure 5As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0131] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0132] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0133] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0134] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0135] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0136] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains an image to be registered of a target area, and first three-dimensional coordinates of a plurality of marking points set in the target area, wherein the first three-dimensional coordinates are determined based on the image to be registered;

[0137] Generating first feature descriptors of the plurality of marking points based on the image to be registered and the first three-dimensional coordinates;

[0138] Based on the first feature descriptor and the second feature descriptors of the multiple marking points, the image to be registered and the reference image of the target area are aligned, wherein the second feature descriptor is generated based on the reference image and the second three-dimensional coordinates of the multiple marking points, and the second three-dimensional coordinates are determined based on the reference image.

[0139] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0140] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0142] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0143] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] The embodiments of the present disclosure also provide a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of any of the above embodiments can be implemented. The execution method and beneficial effects are similar and will not be repeated here.

[0145] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0146] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image registration method, characterized in that: include: Acquire an image to be registered of a target area and first three-dimensional coordinates of a plurality of marking points set in the target area, wherein the first three-dimensional coordinates are determined based on the image to be registered; Generating first feature descriptors of the plurality of marking points based on the image to be registered and the first three-dimensional coordinates; Based on the first feature descriptor and the second feature descriptors of the multiple marking points, the image to be registered and the reference image of the target area are aligned, wherein the second feature descriptor is generated based on the reference image and the second three-dimensional coordinates of the multiple marking points, and the second three-dimensional coordinates are determined based on the reference image.

2. The method according to claim 1, characterized in that Based on the image and the three-dimensional coordinates, generating feature descriptors of the plurality of marker points includes: The multiple marking points are taken as current marking points one by one, and the feature descriptor of the current marking point is generated by the following steps: Based on the image, determine the main direction of the current marker point, wherein the main direction is the direction corresponding to the line connecting the current marker point and a target marker point, the target marker point is one of the surrounding marker points of the current marker point, and the surrounding marker points are marker points other than the current marker point among the multiple marker points; For each of the surrounding marking points except the target marking point, based on the image, determine a preset clockwise angle between the secondary direction of the surrounding marking point and the primary direction, and use it as an angle label value of the surrounding marking point relative to the primary direction, wherein the secondary direction is the direction corresponding to the line connecting the current marking point and the surrounding marking points; Determining an initial feature descriptor of the current marking point based on the angle tag value; Based on the three-dimensional coordinates, the initial feature descriptor is scaled to obtain the feature descriptor of the current marking point; Wherein, when the image is the image to be registered, the feature descriptor is a first feature descriptor, and when the image is the reference image, the feature descriptor is a second feature descriptor.

3. The method according to claim 2, characterized in that The target marking point is the surrounding marking point with the smallest Euclidean distance to the current marking point among all the surrounding marking points.

4. The method according to claim 2, characterized in that: The determining, based on the angle tag value, an initial feature descriptor of the current marking point comprises: For each equally divided sector of the circle with the current marking point as the center, determine all angle label values ​​within the angle interval corresponding to the equally divided sector and add them up to obtain the element corresponding to the equally divided sector; Based on the elements, the initial feature descriptor is determined.

5. The method according to claim 4, characterized in that The step of performing scale correction on the initial feature descriptor based on the three-dimensional coordinates to obtain the feature descriptor of the current marking point includes: Based on the three-dimensional coordinates, obtaining the Euclidean distance from the current marking point to each of the surrounding marking points; Adding the Euclidean distances to obtain a total Euclidean distance, and selecting a minimum value from the Euclidean distances to obtain a minimum Euclidean distance; For each of the surrounding marking points except the target marking point, based on the sum of the Euclidean distances, the minimum Euclidean distance, a preset scale factor and the total number of the plurality of marking points, determine the feature value weight of the surrounding marking point relative to the current marking point, wherein the larger the Euclidean distance between the surrounding marking point and the current marking point, the larger the corresponding feature value weight; For each of the feature value weights, the feature value weight is added to the corresponding element to perform scale correction on the initial feature descriptor to obtain the feature descriptor of the current marking point.

6. The method according to claim 5, characterized in that The step of determining, for each of the surrounding marking points except the target marking point, a feature value weight of the surrounding marking point relative to the current marking point based on the sum of the Euclidean distances, the minimum Euclidean distance, a preset scale factor, and the total number of the plurality of marking points, comprises: For each of the surrounding marking points except the target marking point, the feature value weight of the surrounding marking point relative to the current marking point is calculated using the following formula: Wherein, w is the feature value weight, n is the total number of the multiple marking points, D sum is the sum of the Euclidean distances, D min is the minimum Euclidean distance, β is the preset scale factor, and D is the Euclidean distance from the current marking point to the surrounding marking points.

7. The method according to claim 1, characterized in that The step of aligning the image with a reference image of the target area based on the feature descriptor and the second feature descriptors of the plurality of marking points comprises: Determining a plurality of matching landmark point pairs of the image and the reference image based on the first feature descriptor and the second feature descriptor; Based on the plurality of matching landmark point pairs, a transformation matrix for aligning the image with the reference image is determined.

8. An image registration device, characterized in that: include: An acquisition module, used for acquiring an image to be registered of a target area and first three-dimensional coordinates of a plurality of marking points set in the target area, wherein the first three-dimensional coordinates are determined based on the image to be registered; A generating module, configured to generate first feature descriptors of the plurality of marking points based on the image to be registered and the first three-dimensional coordinates; An alignment module is used to align the image to be registered with a reference image of the target area based on the first feature descriptor and the second feature descriptors of the multiple marking points, wherein the second feature descriptor is generated based on the reference image and the second three-dimensional coordinates of the multiple marking points, and the second three-dimensional coordinates are determined based on the reference image.

9. An electronic device, characterized in that: include: A processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images