Image matching methods, apparatus, devices and media

Abstract

This disclosure relates to an image matching method, apparatus, device, and medium. The method includes: determining a target image patch in a first image corresponding to a first feature point; obtaining a camera rotation matrix of the first image and a gravity vector in the camera coordinate system; determining a horizontal unit vector and a vertical unit vector in the camera coordinate system based on the camera rotation matrix; obtaining a rotation angle of the first image relative to a center point based on the vertical unit vector, the gravity vector, and the horizontal unit vector; determining first feature description information of the first feature point based on the rotation angle and the target image patch; and performing image matching processing on the first image based on the first feature description information. Therefore, by determining the image rotation angle based on the gravity direction, which is independent of the image itself, and determining the extraction direction for extracting feature description information based on the rotation angle, the accuracy and uniqueness of the extracted feature description information are ensured, thus improving the robustness of image matching.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image matching method, apparatus, device and medium. Background Technology

[0002] In scenarios such as visual map positioning, it is common to perform similarity matching between two images. For example, in a map positioning scenario, a scene image is taken, and the captured scene image is matched with a pre-stored image to determine the current location on the map.

[0003] In related technologies, similarity matching between two images is performed based on feature points. The Binary Robust Independent Elementary Features (BRIEF) algorithm selects N point pairs around the feature point to be matched in a certain pattern, and combines the comparison results of these N point pairs as the descriptive information of the feature point. To ensure the rotation invariance of the selected N point pairs, orientation information of the feature point is needed. This orientation information is used as the extraction direction for the extracted point pairs. Traditionally, the gray-scale centroid method is used to determine the orientation information. The gray-scale centroid method uses the image gray values ​​as the center of weights to determine the distance of image patches, and then determines the centroid of the image patch based on the distance. Connecting the geometric center (feature point position) O of the image patch with the centroid C yields the orientation vector. With this direction vector As the extraction direction for N point pairs.

[0004] However, when determining the extraction direction of N point pairs based on the gray value centroid method, the gray values ​​of the image are used as a reference to determine the center of the weights. Therefore, it depends on the gray values ​​of the image itself. If there is too much noise in the image, it will obviously lead to errors in the extraction direction of the N point pairs, resulting in low accuracy of the extracted feature description information and thus affecting the matching accuracy of the image. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides an image matching method, apparatus, device, and medium. The rotation angle of the image is determined based on the direction of gravity, which is independent of the image itself, as a reference. The extraction direction for extracting feature description information is determined based on the rotation angle, which ensures the accuracy and uniqueness of the extracted feature description information and improves the robustness of image matching.

[0006] This disclosure provides an image matching method, the method comprising: determining a target image patch in a first image corresponding to a first feature point, wherein the center point of the target image patch is the first feature point; obtaining a camera rotation matrix of the first image and obtaining a gravity vector in the camera coordinate system; determining a horizontal unit vector and a vertical unit vector in the camera coordinate system based on the camera rotation matrix; obtaining a rotation angle of the first image relative to the center point based on the vertical unit vector, the gravity vector, and the horizontal unit vector; determining first feature description information of the first feature point based on the rotation angle and the target image patch; and performing image matching processing on the first image based on the first feature description information.

[0007] This disclosure also provides an image matching apparatus, comprising: a first determining module, configured to determine a target image patch in a first image corresponding to a first feature point, wherein the center point of the target image patch is the first feature point; a first acquiring module, configured to acquire a camera rotation matrix of the first image and acquire a gravity vector in a camera coordinate system; a second determining module, configured to determine a horizontal unit vector and a vertical unit vector in a camera coordinate system based on the camera rotation matrix; a second acquiring module, configured to acquire a rotation angle of the first image relative to the center point based on the vertical unit vector, the gravity vector, and the horizontal unit vector; a third determining module, configured to determine first feature description information of the first feature point based on the rotation angle and the target image patch; and an image matching module, configured to perform image matching processing on the first image based on the first feature description information.

[0008] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the image matching method provided in this disclosure.

[0009] This disclosure also provides a computer-readable storage medium storing a computer program for performing the image matching method provided in this disclosure.

[0010] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0011] The image matching scheme provided in this embodiment determines a target image patch in a first image corresponding to a first feature point, wherein the center point of the target image patch is the first feature point. It obtains the camera rotation matrix of the first image and the gravity vector in the camera coordinate system. Based on the camera rotation matrix, it determines the horizontal and vertical unit vectors in the camera coordinate system. Then, based on the vertical unit vector, gravity vector, and horizontal unit vector, it obtains the rotation angle of the first image relative to the center point. Based on the rotation angle and the target image patch, it determines the first feature description information of the first feature point. Finally, it performs image matching processing on the first image based on the first feature description information. Therefore, by determining the image rotation angle based on the gravity direction, which is independent of the image itself, and by determining the extraction direction for extracting feature description information based on the rotation angle, it ensures the accuracy and uniqueness of the extracted feature description information and improves the robustness of image matching. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a schematic diagram of an image matching scenario in the prior art provided by an embodiment of the present disclosure;

[0014] Figure 2 This is a schematic diagram of an image matching scenario provided by an embodiment of the present disclosure;

[0015] Figure 3 A flowchart illustrating an image matching method provided in an embodiment of this disclosure;

[0016] Figure 4 This is a schematic diagram of another image matching scenario provided by an embodiment of the present disclosure;

[0017] Figure 5 This is a schematic diagram of another image matching scenario provided by an embodiment of the present disclosure;

[0018] Figure 6 A schematic flowchart of another image matching method provided in an embodiment of this disclosure;

[0019] Figure 7 This is a schematic diagram of another image matching scenario provided by an embodiment of the present disclosure;

[0020] Figure 8 A schematic flowchart of another image matching method provided in an embodiment of this disclosure;

[0021] Figure 9This is a schematic diagram of the structure of an image matching device provided in an embodiment of the present disclosure;

[0022] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0025] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0026] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0027] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0028] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0029] As mentioned above, when determining the extraction direction of descriptors for feature points in an image based on the gray-scale centroid method, it depends on the image itself, and often results in poor matching performance when the image is noisy. Even more so, Figure 1As shown, if there are two symmetrical feature points in the image, the gray-level centroid method cannot distinguish between these two feature points. For example, for two symmetrical feature points A and B, their determined gray-level centroids are Q1 and Q2, respectively. Figure 1 It can be seen that the gray values ​​of the point pair determined by the direction vector from feature point A to Q1 are the same as those of the point pair determined by the direction vector from feature point B to Q2. Therefore, the extracted feature description information of A and B is obviously the same, and it is impossible to distinguish between the two feature points A and B, which further affects the accuracy of image matching.

[0030] To address the aforementioned technical issues and ensure the consistency of the extraction direction of feature description information for extracted feature points, in the embodiments of this disclosure, feature description information extraction no longer relies on the image itself. Instead, it uses the direction of gravity, which is independent of the image itself, to determine the extraction direction of feature description information. This ensures that the extraction direction of feature description information between images is independent of the image itself and is not affected by image noise.

[0031] In addition, such as Figure 2 As shown, in Figure 1 In the scenario shown, when the direction of feature description information extraction is determined based on the direction of gravity as a reference, even if the gray values ​​of the extracted point pairs of symmetrical feature points A and B are different under the direction of gravity, it can be ensured that the corresponding feature description information is different for symmetrical feature points, thus ensuring the accuracy of image matching.

[0032] To address the aforementioned problems, this disclosure provides an image matching method, which will be described below with reference to specific embodiments.

[0033] Figure 3 This is a flowchart illustrating an image matching method provided in an embodiment of the present disclosure. The method can be executed by an image matching device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 3 As shown, the method includes:

[0034] Step 301: Determine the target image block in the first image that corresponds to the first feature point, wherein the center point of the target image block is the first feature point.

[0035] The first image can be any image among the images to be matched. Since image matching in this embodiment is based on feature points as the matching object, this disclosure primarily discloses a method for obtaining feature description information of the first feature point in the first image. The first feature point can be any feature point in the first image.

[0036] In this embodiment, based on the feature registration principle of algorithms such as BRIEF, in order to obtain the feature description information of the first feature point, the target image block corresponding to the first feature point is obtained, wherein the center point of the target image block is the first feature point.

[0037] In some possible embodiments, a target image block is determined in the first image with a first feature point as the center, based on a preset size and a preset shape.

[0038] The aforementioned preset size and preset shape can be calibrated according to the scene, and the preset size is used to limit the size of the preset shape.

[0039] In actual execution, to ensure rotation consistency, the preset shape can be circular. Regardless of how the circular target image block rotates around its center, the preceding and following circular target image blocks will always completely overlap. This guarantees that, for the same feature point, the pixels involved in calculating feature description information are exactly the same after the target image block has rotated.

[0040] Step 302: Obtain the camera rotation matrix of the first image and the gravity vector in the camera coordinate system.

[0041] The gravity vector can be understood as the direction vector of gravity in the world coordinate system in the camera coordinate system.

[0042] It is easy to understand that the direction of gravity is fixed. Regardless of the shooting angle of the image or the camera coordinate system, the direction of gravity remains unchanged in the world coordinate system. Therefore, in the embodiments of this disclosure, the gravity vector in the camera coordinate system is determined with reference to the direction of gravity, and the rotation angle of the image is further determined based on this gravity vector.

[0043] In some possible embodiments, gravity direction information in the world coordinate system can be obtained. This gravity direction information can be obtained by a sensor in the camera device that reads the first image. Then, based on the camera rotation matrix and the gravity direction information, the gravity vector in the camera coordinate system under the action of the camera rotation matrix is ​​determined.

[0044] Step 303: Determine the horizontal and vertical unit vectors in the camera coordinate system based on the camera rotation matrix.

[0045] The camera rotation matrix can be understood as the extrinsic parameter of the camera that takes the first image. The camera rotation matrix is ​​a 3*3 rotation matrix. This rotation matrix is ​​used to transform feature points in the world coordinate system to the camera coordinate system. Therefore, based on the camera rotation coordinates, we can obtain the horizontal unit vector (X-axis direction) and vertical unit vector (Z-axis direction) of the coordinate system in the world coordinate system due to the influence of the rotation angle when the camera is shooting.

[0046] In one embodiment of this disclosure, horizontal rotation parameters and vertical rotation parameters corresponding to the camera rotation matrix are determined, a horizontal unit vector is determined based on the horizontal rotation parameters, and a vertical unit vector is determined based on the vertical rotation parameters.

[0047] The rotation parameter reflects the transformation from the world coordinate system to the camera coordinate system. Therefore, the unit vector in the camera coordinate system can be determined based on the rotation parameter.

[0048] For example, if the rotation angle of the camera coordinate system relative to the world coordinate system on the X-axis is α, then its rotation matrix on the X-axis is as shown in formula (1) below. If the rotation angle of the camera coordinate system relative to the world coordinate system on the Y-axis is β, then its rotation matrix on the Y-axis is as shown in formula (2) below. If the rotation angle of the camera coordinate system relative to the world coordinate system on the Z-axis is θ, then its rotation matrix on the Z-axis is as shown in formula (3) below.

[0049]

[0050]

[0051]

[0052] The camera rotation matrix is ​​then given by the following formula (4):

[0053]

[0054] like The horizontal unit vector can then be determined as Define the vertical unit vector as

[0055] Step 304: Obtain the rotation angle of the first image relative to the center point based on the vertical unit vector, gravity vector, and horizontal unit vector.

[0056] In one embodiment of this disclosure, since the vertical unit vector and the horizontal unit vector represent the actual presentation results of the vertical direction vector and the horizontal direction vector of the world coordinate system in the camera coordinate system, and the gravity vector represents the theoretical presentation result of the vertical direction vector of the world coordinate system in the camera coordinate system, the rotation angle of the first image relative to the gravity direction can be determined based on the comparison between the actual presentation result and the theoretical presentation result.

[0057] Since the process of determining the rotation angle is independent of the grayscale value of the target image block itself, the rotation angle of the first image relative to the center point is obtained based on the vertical unit vector, gravity vector, and horizontal unit vector. This will not be affected by some parameters of the image itself, such as the noise of the image itself, thus ensuring the consistency of the rotation angle between images during image matching.

[0058] Step 305: Determine the first feature description information of the first feature point based on the rotation angle and the target image block, and perform image matching processing on the first image based on the first feature description information.

[0059] In one embodiment of this disclosure, after determining the rotation angle, the first feature description information of the first feature point is determined based on the rotation angle and the target image block. The first feature description information is used to describe the first feature point and is used as a feature of the first feature point to perform feature matching with feature points of other images. The rotation angle in this embodiment can be used as the extraction direction of N pairs of points in the above-mentioned BRIEF algorithm.

[0060] In this embodiment, as Figure 4 As shown, if the first feature point is 'a' and the target image block is rectangular, then the initial coordinate system of the target image block is OXY (the original direction of the target image block is the solid line direction). After determining the gravity vector OG, the rotation angle calculated based on OG is γ. Based on γ, the target image block is rotated to the position shown by the dashed line. At this time, the coordinate system of the rotated target image block is OXγG. Since the gravity direction of OXγG is fixed, the feature description information of the first feature point can be extracted based on the coordinate system under this direction, which can ensure the consistency of rotation.

[0061] When extracting the feature description information of the first feature point, any BRIEF algorithm can be used to extract N pairs of points (N can be 218, 516, etc., which are not listed here). In one embodiment of this disclosure, the original image coordinate system of the target image block can be corrected according to the rotation angle to obtain the target image coordinate system. Under the target image coordinate system, with the center point as the origin, multiple pairs of pixels in the target image block are extracted according to a preset extraction strategy. The pixel value of each pair of pixels is extracted, and the comparison result information of each pair of feature points is determined according to the pixel value. The first feature description information of the first feature point is determined according to the multiple comparison result information corresponding to multiple pairs of pixels.

[0062] For example, such as Figure 5 As shown, each pixel can be numbered according to the coordinate system after rotation by the rotation angle (each circle in the figure marks a pixel). Pixel pairs are extracted based on the extraction mode. Based on the comparison result of the gray values ​​of the pixel pairs (for example, if the gray value of 12 is greater than that of 119 among the two extracted pixels 12 and 119, the comparison result is determined to be 1; if the gray value of 12 is less than that of 119, the comparison result is determined to be 0), the first feature description information of the first feature point is determined based on the comparison results of N point pairs.

[0063] It is easy to understand that when we observe an object from different distances, directions, angles, and lighting conditions, the object's size, shape, and brightness distribution will differ. However, our brains can still perceive it as an object. Ideally, feature description information should also behave similarly. That is, for two images with different sizes, orientations, and grayscale brightness changes, the same feature point should have similar feature description information. Therefore, in the embodiments of this disclosure, while ensuring the spatial consistency of the pixel pairs extracted from the first feature description information, the reliability of image matching processing of the first image based on the first feature description information can be guaranteed. Furthermore, when extracting feature description information based on the direction of gravity, even symmetrical feature points under the direction of gravity will have different grayscale values. Therefore, for symmetrical feature points, it can also be ensured that their corresponding feature description information is different, further guaranteeing the accuracy of image matching.

[0064] In summary, the image matching method of this disclosure identifies a target image patch in a first image corresponding to a first feature point, wherein the center point of the target image patch is the first feature point. It obtains the camera rotation matrix of the first image and the gravity vector in the camera coordinate system. Based on the camera rotation matrix, it determines the horizontal and vertical unit vectors in the camera coordinate system. Then, based on the vertical unit vector, gravity vector, and horizontal unit vector, it obtains the rotation angle of the first image relative to the center point. Based on the rotation angle and the target image patch, it determines the first feature description information of the first feature point. Finally, it performs image matching processing on the first image based on the first feature description information. Therefore, by using the gravity direction, which is independent of the image itself, as a reference to determine the image rotation angle, and by determining the extraction direction for extracting feature description information based on the rotation angle, the accuracy and uniqueness of the extracted feature description information are ensured, thus improving the robustness of image matching.

[0065] It should be noted that the method for obtaining the rotation angle of the first image relative to the center point based on the vertical unit vector, gravity vector, and horizontal unit vector differs in different application scenarios, as shown in the following example:

[0066] In one embodiment of this disclosure, a deep learning model is pre-trained based on a large amount of sample data. The vertical unit vector, gravity vector, and horizontal unit vector are input into the trained deep learning model to obtain the rotation angle output by the deep learning model.

[0067] In one embodiment of this disclosure, such as Figure 6 As shown, the rotation angle of the first image relative to the center point is obtained based on the vertical unit vector, the gravity vector, and the horizontal unit vector, including:

[0068] Step 601: Determine the rotation reference vector based on the vertical unit vector and the gravity vector.

[0069] like Figure 7 As shown, it can be understood that the vertical unit vector can be regarded as the actual unit vector and ideal unit vector in the Z-axis direction of the world coordinate system in the camera coordinate system (with the first feature point as the center of rotation, it is necessary to ensure the consistency of the unit vector on the Z-axis). The rotation reference vector can be regarded as the actual unit vector in the X-axis direction of the world coordinate system in the camera coordinate system, and the gravity vector can be regarded as the ideal unit vector in the Y-axis direction of the gravity direction in the world coordinate system in the camera coordinate system. Therefore, based on the structure of the coordinate system, the rotation reference vector (ideal unit vector in the X-axis) can be determined based on the vertical unit vector (ideal unit vector in the Z-axis) and the gravity vector (ideal unit vector in the Y-axis).

[0070] In some possible embodiments, the vector product of the vertical unit vector and the gravity vector is calculated, and the vector product is determined to be the rotation reference vector, wherein if the vertical unit vector is... The gravity vector is Rotation reference vector is Then it can be calculated using the following formula (5):

[0071]

[0072] Step 602: Obtain the rotation angle based on the rotation reference vector and the horizontal unit vector.

[0073] In this embodiment, after determining the rotation reference vector, since the rotation reference vector is the ideal unit vector on the X-axis when the world coordinate system is converted to the camera coordinate system, and the horizontal unit vector is the actual unit vector on the X-axis in the world coordinate system under the camera coordinate system, the rotation angle of the current display angle of the target image block in the world coordinate system can be obtained based on the rotation reference vector and the horizontal unit vector.

[0074] In some possible embodiments, the inner and outer products of the rotation reference vector and the horizontal unit vector can be calculated separately, the ratio of the outer product to the inner product can be calculated, and the arcsine function of the ratio can be calculated to obtain the rotation angle. For example, if the rotation reference vector is... The horizontal unit vector is And refer to Figure 5 It can be seen that, Therefore, the rotation angle γ can be calculated using the following formula (6).

[0075]

[0076] In other possible embodiments, based on the principle of image rotation, the inner and outer products of the gravity direction vector and the vertical unit vector can also be calculated separately, where the vertical unit vector can be one of the components in the aforementioned camera rotation matrix. Calculate the inner and outer products of the gravity direction vector and the vertical unit vector, respectively. Calculate the ratio of the outer and inner products, and then calculate the arcsine function of the ratio to obtain the rotation angle.

[0077] In summary, the image matching method of this disclosure can use the gravity vector in the camera coordinate system as a reference to determine the rotation angle of the first image. By using parameters that are independent of the image itself to determine the rotation angle, the rotation invariance of similar feature points during image matching is ensured, thereby improving the accuracy of image matching.

[0078] Based on the above embodiments, after determining the first feature description information of the first feature point according to the rotation angle and the target image block, since the first feature information is extracted with the gravity direction as the reference direction, and since the gravity direction is defined in the world coordinate system, the rotation direction determined based on the gravity direction can ensure the rotation consistency of the feature point's position in space during image matching. For example, we have a stamp with some straight lines on it. When we stamp an image with this stamp, a certain point on the image is extracted. If the stamp remains unchanged, and we rotate the image and stamp it again, the extracted point will be different from the previous one. To make the two extracted points the same, we need to rotate the stamp by an angle. In this embodiment, the purpose of the rotation angle is to rotate this "stamp". Therefore, after the rotation angle is determined, the feature description information extracted with this rotation angle as the extraction angle can ensure rotation invariance and improve the accuracy of image matching. The following is a detailed description of the image matching process:

[0079] In one embodiment of this disclosure, such as Figure 8 As shown, image matching processing of the first image based on the first feature description information includes:

[0080] Step 801: Obtain the second feature description information for each reference feature point in the second image.

[0081] The method for determining the second feature description information is the same as the method for determining the first feature description information. In one embodiment of this disclosure, a reference image block corresponding to each reference feature point in the second image is determined, wherein the center point of the reference image block is the corresponding reference feature point. The reference camera rotation matrix of the reference image block is obtained, and the reference gravity vector in the camera coordinate system is obtained. The reference horizontal unit vector and the reference vertical unit vector in the world coordinate system are determined according to the reference camera rotation matrix. The rotation angle of the second image is determined according to the reference vertical unit vector, the reference gravity vector, and the reference horizontal unit vector. The second feature description information of the corresponding reference feature point is obtained according to the rotation angle and the reference image block.

[0082] Since both the second feature description information and the first feature description information are extracted based on the direction of gravity as the rotation angle, the second feature description information and the first feature description information have rotation invariance. If the reference feature point and the first feature point are similar feature points in the same space, then the first feature description information and the second feature description information are similar and will not be affected by the different rotation angles of the image.

[0083] Step 802: Based on the first feature description information and the second feature description information, determine the second feature point in the second image that matches the first feature point.

[0084] In this embodiment, based on the feature point matching principle, a second feature point in the second image that matches the first feature point is determined according to the first feature description information and the second feature description information. Since the first feature description information and the second feature description information are extracted in the same spatial direction, if the second feature description information of the second feature point matches the first feature description information, then the first feature point and the second feature point belong to similar feature points in the same spatial location.

[0085] Step 803: Perform image matching processing on the first image and the second image based on the first feature point and the second feature point.

[0086] The method of image matching processing based on the first feature point and the second feature point can vary in different application scenarios:

[0087] In one embodiment of this disclosure, whether the first image and the second image match can be determined based on the number of matching first feature points and second feature points, etc.

[0088] In one embodiment of this disclosure, whether the first image and the second image match can be determined based on the matching weights of the first feature point and the second feature point.

[0089] In summary, the image matching method of this disclosure determines the transformation angle of the image with the gravity direction in the world coordinate system as a reference, and uses the rotation angle as the extraction direction of descriptive feature information during image matching, thereby ensuring the rotation invariance of the feature description information during image matching and improving the reliability of image matching.

[0090] To implement the above embodiments, this disclosure also proposes an image matching device.

[0091] Figure 9 This is a schematic diagram of an image matching device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device for image matching. Figure 9 As shown, the device includes: a first determining module 910, a first acquiring module 920, a second determining module 930, a second acquiring module 940, a third determining module 950, and an image matching module 960, wherein,

[0092] The first determining module 910 is used to determine a target image block in the first image corresponding to the first feature point, wherein the center point of the target image block is the first feature point;

[0093] The first acquisition module 920 is used to acquire the camera rotation matrix of the first image and the gravity vector in the camera coordinate system;

[0094] The second determining module 930 is used to determine the horizontal unit vector and the vertical unit vector in the world coordinate system according to the camera rotation matrix.

[0095] The second acquisition module 940 is used to acquire the rotation angle of the first image relative to the center point based on the vertical unit vector, the gravity vector and the horizontal unit vector;

[0096] The third determining module 950 is used to determine the first feature description information of the first feature point based on the rotation angle and the target image block;

[0097] The image matching module 960 is used to perform image matching processing on the first image based on the first feature description information.

[0098] The image matching apparatus provided in this disclosure can execute the image matching method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0099] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program / instructions that, when executed by a processor, implement the image matching method described above.

[0100] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0101] The following is a detailed reference. Figure 10 The diagram illustrates a structural schematic suitable for implementing the electronic device 1000 in the embodiments of this disclosure. The electronic device 1000 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0102] like Figure 10As shown, the electronic device 1000 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a memory 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processor 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0103] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; memory devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0104] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from memory 1008, or installed from ROM 1002. When the computer program is executed by processor 1001, it performs the functions defined in the image matching method of embodiments of this disclosure.

[0105] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0106] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0107] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0108] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0109] A target image patch corresponding to a first feature point in the first image is identified, where the center point of the target image patch is the first feature point. The camera rotation matrix of the first image is obtained, along with the gravity vector in the camera coordinate system. Based on the camera rotation matrix, the horizontal and vertical unit vectors in the world coordinate system are determined. Then, based on the vertical unit vector, gravity vector, and horizontal unit vector, the rotation angle of the first image relative to the center point is obtained. The first feature description information of the first feature point is determined based on the rotation angle and the target image patch. Image matching processing is then performed on the first image based on the first feature description information. Thus, by using the gravity direction, which is independent of the image itself, as a reference to determine the image rotation angle, and by using the rotation angle to determine the extraction direction for extracting feature description information, the accuracy and uniqueness of the extracted feature description information are ensured, improving the robustness of image matching.

[0110] Electronic devices can be programmed with computer program code in one or more programming languages ​​or combinations thereof to perform the operations of this disclosure. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0113] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0114] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0115] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0116] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0117] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image matching method, characterized in that, Includes the following steps: Determine a target image block in the first image that corresponds to a first feature point, wherein the center point of the target image block is the first feature point; Obtain the camera rotation matrix of the first image and the gravity vector in the camera coordinate system; The horizontal and vertical unit vectors in the camera coordinate system are determined based on the camera rotation matrix. A rotation reference vector is determined based on the vertical unit vector and the gravity vector, and the rotation angle of the first image relative to the center point is obtained based on the rotation reference vector and the horizontal unit vector. Based on the rotation angle and the target image block, the first feature description information of the first feature point is determined, and the first image is subjected to image matching processing based on the first feature description information.

2. The method as described in claim 1, characterized in that, Determining the target image patch in the first image corresponding to the first feature point includes: Centered on the first feature point, the target image block is determined in the first image according to a preset size and preset shape.

3. The method as described in claim 1, characterized in that, The process of obtaining the gravity vector in the camera coordinate system includes: Obtain gravity direction information in the world coordinate system; Based on the camera rotation matrix and the gravity direction information, the gravity vector in the camera coordinate system is obtained.

4. The method as described in claim 1, characterized in that, The step of determining the horizontal and vertical unit vectors in the camera coordinate system based on the camera rotation matrix includes: Determine the horizontal and vertical rotation parameters corresponding to the camera rotation matrix; The horizontal unit vector is determined based on the horizontal rotation parameters; The vertical unit vector is determined based on the vertical rotation parameters.

5. The method as described in claim 1, characterized in that, The step of determining the rotation reference vector based on the vertical unit vector and the gravity vector includes: Calculate the vector product of the vertical unit vector and the gravity vector; The vector product is determined to be the rotation reference vector.

6. The method as described in claim 1 or 5, characterized in that, The step of obtaining the rotation angle based on the rotation reference vector and the horizontal unit vector includes: Calculate the inner product and outer product of the rotation reference vector and the horizontal unit vector, respectively; Calculate the ratio of the outer product to the inner product, and calculate the arcsine function of the ratio to obtain the rotation angle.

7. The method as described in claim 1, characterized in that, The step of determining the first feature description information of the first feature point based on the rotation angle and the target image block includes: The original image coordinate system of the target image block is corrected according to the rotation angle to obtain the target image coordinate system; In the target image coordinate system, with the center point as the origin, multiple pairs of pixels in the target image block are extracted according to a preset extraction strategy; Extract the pixel value of each pair of pixels, and determine the comparison result information of each pair of feature points based on the pixel value; The first feature description information of the first feature point is determined based on the comparison result information corresponding to the multiple pairs of pixels.

8. The method as described in claim 1, characterized in that, The step of performing image matching processing on the first image based on the first feature description information includes: Obtain the second feature description information for each reference feature point in the second image; Based on the first feature description information and the second feature description information, determine the second feature point in the second image that matches the first feature point; Image matching processing is performed on the first image and the second image based on the first feature point and the second feature point.

9. The method as described in claim 8, characterized in that, The step of obtaining the second feature description information for each reference feature point in the second image includes: Determine a reference image block in the second image corresponding to each reference feature point, wherein the center point of the reference image block is the corresponding reference feature point; Obtain the reference camera rotation matrix of the reference image block, and obtain the reference gravity vector in the camera coordinate system; The reference horizontal unit vector and the reference vertical unit vector in the camera coordinate system are determined based on the reference camera rotation matrix. The rotation angle of the second image is determined based on the reference vertical unit vector, the reference gravity vector, and the reference horizontal unit vector. The second feature description information of the corresponding reference feature points is obtained based on the rotation angle and the reference image block.

10. An image matching device, characterized in that, include: A first determining module is used to determine a target image block in a first image corresponding to a first feature point, wherein the center point of the target image block is the first feature point; The first acquisition module is used to acquire the camera rotation matrix of the first image and the gravity vector in the camera coordinate system; The second determining module is used to determine the horizontal unit vector and the vertical unit vector in the camera coordinate system according to the camera rotation matrix; The second acquisition module is used to determine a rotation reference vector based on the vertical unit vector and the gravity vector, and to acquire the rotation angle of the first image relative to the center point based on the rotation reference vector and the horizontal unit vector. The third determining module is used to determine the first feature description information of the first feature point based on the rotation angle and the target image block; The image matching module is used to perform image matching processing on the first image based on the first feature description information.

11. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the image matching method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for performing the image matching method according to any one of claims 1-9.

Images