An image-based map fusion method and device

Abstract

An embodiment of the present application discloses an image-based map fusion method and device. The method includes: matching any first image collected by a first device with at least one second image collected by a second device respectively to determine at least one similarity; if there is a target similarity greater than a preset similarity threshold among them, determining a target first image and a target second image corresponding to the target similarity; determining a first heading angle of the first device at a first moment according to the pose information of the first device at the first moment, and determining a second heading angle of the second device at a second moment according to the pose information of the second device at the second moment; if the absolute value of the difference between the first heading angle and the second heading angle is less than a preset angle threshold, determining that the poses of the first device and the second device are the same, and fusing the maps corresponding to the first device and the second device respectively based on the pose information of the first device and the second device. The accuracy of map fusion is improved.

Description

Technical Field

[0001] This application relates to the field of map fusion technology, and in particular, to a method and device for map fusion based on images. Background Art

[0002] Visual SLAM (Simultaneous Localization and Mapping) means that a target carrier is equipped with a visual camera and moves in an unknown environment. By observing the surrounding environment, it determines its own position and attitude, and then builds a map of the surrounding environment based on its own position, realizing the localization of the target carrier and the construction of the surrounding environment. Based on the continuous growth of current positioning technology requirements, visual SLAM technology is widely used in fields such as driverless cars, mobile robots, and unmanned logistics vehicles.

[0003] During the map construction process, multiple SLAM systems often separately construct maps and then fuse the maps they have constructed to form a panoramic map. In related technologies, during the SLAM map fusion process, it is usually determined whether to fuse the maps constructed by each according to the similarity of images. In this method, when two frames of images are collected at different spatial positions but have a similar scene, making a judgment solely based on image information may misjudge that the two frames of images are collected at the same location, resulting in misjudgment. Consequently, the fused map is inaccurate. Summary of the Invention

[0004] Embodiments of this application provide a method and device for map fusion based on images to improve the accuracy of map fusion.

[0005] In a first aspect, an embodiment of this application provides a method for map fusion based on images, including:

[0006] For any first image collected by a first device, match it with at least one second image collected by a second device respectively to determine at least one similarity;

[0007] If there is a target similarity greater than a preset similarity threshold among the at least one similarity, determine the target first image and the target second image corresponding to the target similarity;

[0008] Determine the first heading angle of the first device at the first moment according to the pose information of the first device at the first moment, and determine the second heading angle of the second device at the second moment according to the pose information of the second device at the second moment; where the first moment is the acquisition moment of the target first image, and the second moment is the acquisition moment of the target second image;

[0009] If the absolute value of the difference between the first heading angle and the second heading angle is less than a preset angle threshold, it is determined that the poses of the first device and the second device are the same, and the maps corresponding to the first device and the second device are fused based on the pose information of the first device and the second device.

[0010] In a second aspect, an embodiment of the present application provides an image-based map fusion device, including a processor, a memory, and at least one external communication interface, and the processor, the memory, and the external communication interface are all connected through a bus;

[0011] The external communication interface is configured to receive any one first image collected by a first device and at least one second image collected by a second device;

[0012] A computer program is stored in the memory, and the processor is configured to perform the following operations based on the computer program:

[0013] For any one first image collected by the first device, match it with at least one second image collected by the second device respectively to determine at least one similarity;

[0014] If there is a target similarity greater than a preset similarity threshold among the at least one similarity, determine the target first image and the target second image corresponding to the target similarity;

[0015] Determine the first heading angle of the first device at the first moment according to the pose information of the first device at the first moment, and determine the second heading angle of the second device at the second moment according to the pose information of the second device at the second moment; wherein, the first moment is the acquisition moment of the target first image, and the second moment is the acquisition moment of the target second image;

[0016] If the absolute value of the difference between the first heading angle and the second heading angle is less than a preset angle threshold, it is determined that the poses of the first device and the second device are the same, and the maps corresponding to the first device and the second device are fused based on the pose information of the first device and the second device.

[0017] In a third aspect, an embodiment of the present application provides an apparatus for image-based map fusion, including:

[0018] A similarity determination module, configured to match any one first image collected by a first device with at least one second image collected by a second device respectively to determine at least one similarity;

[0019] A target image determination module, configured to determine a target first image and a target second image corresponding to the target similarity when there is a target similarity greater than a preset similarity threshold among the at least one similarity;

[0020] A heading angle determination module, configured to determine a first heading angle of the first device at the first moment according to the pose information of the first device at the first moment, and determine a second heading angle of the second device at the second moment according to the pose information of the second device at the second moment; wherein, the first moment is the acquisition moment of the target first image, and the second moment is the acquisition moment of the target second image;

[0021] A fusion module, configured to determine that the poses of the first device and the second device are the same when the absolute value of the difference between the first heading angle and the second heading angle is less than a preset angle threshold, and fuse the maps corresponding to the first device and the second device respectively based on the pose information of the first device and the second device.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of any of the above methods are implemented.

[0023] The embodiments of the present application have the following beneficial effects:

[0024] By respectively matching any first image collected by the first device with at least one second image collected by the second device to determine at least one similarity, when there is a target similarity greater than a preset similarity threshold, the target first image and the target second image corresponding to the target similarity are determined. In this way, first, the similarity matching is used to determine that the first device and the second device are in a similar scene. Then, the first heading angle of the first device at the first moment is determined according to the pose information of the first device at the first moment, and the second heading angle of the second device at the second moment is determined according to the pose information of the second device at the second moment. When the absolute value of the difference between the first heading angle and the second heading angle is less than a preset angle threshold, it is determined that the poses of the first device and the second device are the same, and the maps corresponding to the first device and the second device are fused based on the pose information of the first device and the second device respectively. In this way, the heading angle is used to determine that the first device and the second device are in the same spatial position. It can largely prevent the occurrence of misjudgment in similar scenes in different spaces and improve the accuracy of map fusion. Description of the Drawings

[0025] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for the embodiments of the present application will be briefly introduced below. Obviously, the accompanying drawings introduced below are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0026] Figure 1 It is a schematic diagram of an application scenario of an image-based map fusion method provided by an embodiment of the present application;

[0027] Figure 2 It is a schematic diagram of a situation where scenes are similar in different spaces provided by an embodiment of the present application;

[0028] Figure 3 It is a schematic flowchart of an image-based map fusion method provided by an embodiment of the present application;

[0029] Figure 4 It is a schematic diagram of the pose of a SLAM system provided by an embodiment of the present application;

[0030] Figure 5 It is a schematic flowchart of another image-based map fusion method provided by an embodiment of the present application;

[0031] Figure 6 It is a schematic structural diagram of an image-based map fusion device provided by an embodiment of the present application;

[0032] Figure 7 It is a schematic structural diagram of an image-based map fusion device provided by an embodiment of the present application. Detailed implementation manners

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application.

[0034] The number of any element in the accompanying drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0035] In the actual map construction process, it often happens that multiple SLAM systems separately construct maps and then fuse the maps they have constructed respectively to form a panoramic map. In the related art, the common practice is to decide whether to fuse the maps constructed respectively based on the similarity of images during the SLAM map fusion process. In this method, when two frames of images are captured at different spatial positions but have similar scenes, making a judgment solely based on image information may mistakenly think that the two frames of images are captured at the same location, and misjudge that the two SLAM systems are at the same spatial position. As a result, the fused map is inaccurate.

[0036] Therefore, this application provides an image-based map fusion method, which is applicable to the situation of map fusion of images captured by two devices. By respectively matching any first image captured by the first device with at least one second image captured by the second device, at least one similarity is determined. First, based on the magnitude relationship between at least one similarity and a preset similarity threshold, the target first image and the target second image corresponding to the target similarity greater than the preset similarity threshold are determined. Further, the difference between the first heading angle of the first device at the acquisition moment of the target first image and the second heading angle of the second device at the acquisition moment of the target second image is compared. If the absolute value of the difference between the two is less than the preset angle threshold, it can be determined that the poses of the first device and the second device are the same. Then, based on the position information of the first device and the second device, the maps corresponding to the first device and the second device are fused respectively. Compared with the related art that only determines the same pose of two devices based on similarity, the heading information of the two devices is fused, solving the problem of incorrect map fusion in similar scenes under different scenarios.

[0037] After introducing the design concept of the embodiments of this application, the following briefly introduces the application scenarios applicable to the technical solutions of the embodiments of this application. It should be noted that the following introduced application scenarios are only for illustrating the embodiments of this application rather than limiting them. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0038] Reference Figure 1, which is a schematic diagram of an application scenario of an image-based map fusion method provided by an embodiment of the present application. The first device 11 and the second device 12 are, for example, devices integrated with a SLAM system, such as intelligent robots or AR devices, etc. Taking an intelligent robot as an example, in this example, assuming that it is uncertain whether Room 1 and Room 2 are the same space, two different spaces are used to show. The intelligent robot 1 and the intelligent robot 2 move in Room 1 and Room 2 respectively. For the intelligent robot 1, the intelligent robot 1 moves in Room 1, determines its own position and pose by observing Room 1, and then builds a map of Room 1 according to its own position, realizing the positioning and environment construction of Room 1. For the intelligent robot 2, the intelligent robot 2 moves in Room 2, determines its own position and pose by observing Room 2, and then builds a map of Room 2 according to its own position, realizing the positioning and environment construction of Room 2.

[0039] In a specific example, Figure 2 shows a schematic diagram of a situation where the scenes are similar in different spaces. For example, both Room 21 and Room 22 include a conference table and 4 conference chairs. The orientation of the room door is determined as the room orientation. In this example, the orientation of Room 21 is south, and the orientation of Room 2 is west. At this time, the heading angle information can be introduced to determine whether the poses of the intelligent robots in these two rooms are the same. If they are the same, the maps corresponding to the first device and the second device can be fused based on the pose information of the first device and the pose information of the second device.

[0040] Of course, the method provided by the embodiment of the present application is not limited to Figure 1 the application scenario shown, and can also be used in other possible application scenarios, which are not limited by the embodiment of the present application. For Figure 1 the functions that can be realized by each device in the application scenario shown will be described together in the subsequent method embodiments, and will not be elaborated here too much.

[0041] To further illustrate the technical solution provided by the embodiment of the present application, the following will be described in detail in combination with the accompanying drawings and specific implementation manners. Although the embodiment of the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or non-creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application.

[0042] The following will be combined with Figure 1 the application scenario shown to illustrate the technical solution provided by the embodiment of the present application.

[0043] Refer to Figure 3, an embodiment of the present application provides an image-based map fusion method, including the following steps:

[0044] S301. For any first image collected by the first device, respectively match it with at least one second image collected by the second device to determine at least one similarity.

[0045] S302. If there is a target similarity greater than a preset similarity threshold among the at least one similarity, determine the target first image and the target second image corresponding to the target similarity.

[0046] S303. Determine the first heading angle of the first device at the first moment according to the pose information of the first device at the first moment, and determine the second heading angle of the second device at the second moment according to the pose information of the second device at the second moment; wherein, the first moment is the acquisition moment of the target first image, and the second moment is the acquisition moment of the target second image.

[0047] S304. If the absolute value of the difference between the first heading angle and the second heading angle is less than a preset angle threshold, determine that the poses of the first device and the second device are the same, and fuse the maps corresponding to the first device and the second device respectively based on the pose information of the first device and the second device.

[0048] In the embodiment of the present application, by respectively matching any first image collected by the first device with at least one second image collected by the second device to determine at least one similarity, when there is a target similarity greater than a preset similarity threshold, determine the target first image and the target second image corresponding to the target similarity. In this way, first, determine that the first device and the second device are in a similar scene through similarity matching. Then, determine the first heading angle of the first device at the first moment according to the pose information of the first device at the first moment, and determine the second heading angle of the second device at the second moment according to the pose information of the second device at the second moment. When the absolute value of the difference between the first heading angle and the second heading angle is less than a preset angle threshold, determine that the poses of the first device and the second device are the same, and fuse the maps corresponding to the first device and the second device respectively based on the pose information of the first device and the second device. In this way, then determine that the first device and the second device are in the same spatial position through the heading angle. It can largely prevent the occurrence of misjudgment in similar scenes in different spaces and improve the accuracy of map fusion.

[0049] Regarding S201, the first device and the second device move in their respective acquisition spaces through the vision cameras mounted on themselves, and collect environmental images in their respective acquisition spaces in real time. The map fusion device in the embodiments of the present application is, for example, a server or a computer with map fusion function, and receives in real time any first image collected by the first device and at least one second image collected by the second device; then, for any first image collected by the first device, it is respectively matched with at least one second image collected by the second device to determine at least one similarity.

[0050] In a specific example, for instance, within a preset time period, the first device collects 70 first images and the second device collects 50 second images, then 70 * 50 = 3500 similarity values can be determined. However, in the actual application process, the image acquisition process is real-time, and the similarity determination process is also real-time. Therefore, it is not limited to calculating 3500 similarities after collecting 3500 images.

[0051] In the actual application process, the similarity between any first image and any second image can be determined in the following manner:

[0052] For any first image collected by the first device, extract the image feature points of the first image and integrate the feature points of the first image into a first bag-of-words vector; and, for any second image collected by the second device, extract the image feature points of the second image and integrate the feature points of the second image into a second bag-of-words vector; determine the similarity between the first image and the second image according to the distance between the first bag-of-words vector and the second bag-of-words vector.

[0053] Among them, through the bag-of-words model, based on the feature points of the image, an image can be described by a bag-of-words vector, and then the similarity between the corresponding two images is determined by calculating the distance between the two bag-of-words vectors. Therefore, in the embodiments of the present application, the above method can be used to determine the similarity between any first image and any second image.

[0054] Regarding 302, for each obtained similarity, compare its size relationship with the preset similarity threshold. If there is a target similarity greater than the preset similarity threshold, determine the target first image and the target second image corresponding to the target similarity. In a specific example, the preset similarity threshold can be determined according to actual needs, such as 0.95. The larger the preset similarity threshold, the higher the accuracy of map fusion.

[0055] In the actual application process, the comparison process between each similarity and the preset similarity threshold can be a traversal process. That is, when the first similarity is not greater than the preset similarity threshold, the second similarity is compared until the target similarity greater than the preset similarity threshold is found, and then subsequent processing is performed on the corresponding target first image and target second image.

[0056] Regarding S303, to prevent the problem that the visual sensors of the first device and the SLAM coordinate system of the first device have the same view in different spaces due to angle reasons, a coordinate axis comparison will be performed. This comparison process is achieved by comparing the first heading angle of the first device at the first moment with the heading angle of the second device at the second moment.

[0057] After determining the target first image and target second image corresponding to the target similarity greater than the threshold, the timestamp information of the target first image can be extracted and recorded as the first moment, which is the acquisition moment of the target first image; the timestamp information of the target second image is extracted and recorded as the second moment, which is the acquisition moment of the target second image, so that when determining the heading angle, the corresponding moment is used as a reference to determine the first heading angle of the first device at the first moment and the second heading angle of the second device at the second moment.

[0058] The pose information of the first device at the first moment is obtained through the AHRS system of the first device, and the pose information of the second device at the second moment is obtained through the AHRS system of the second device. Then, the first heading angle of the first device at the first moment is determined according to the pose information of the first device at the first moment, and the second heading angle of the second device at the second moment is determined according to the pose information of the second device at the second moment.

[0059] Exemplarily, the determination process of the first heading angle of the first device at the first moment is as follows:

[0060] A. Determine the first magnetic heading angle of the first device at the first moment in the geomagnetic coordinate system according to the pose information of the first device at the first moment.

[0061] B. Apply the first magnetic heading angle and the magnetic declination of the area where the first device is located to determine the first heading angle of the first device at the first moment.

[0062] First, the determination process of the first magnetic heading angle is described, which mainly includes the following two steps, A1 and A2:

[0063] A1. Determine the target horizontal axis magnetic component and target vertical axis magnetic component of the first device in the geomagnetic coordinate system at the first moment according to the three-axis initial magnetic components of the magnetometer of the first device at the first moment obtained, the roll angle and pitch angle of the first device at the first moment.

[0064] Among them, the pose information is determined by, for example, obtaining the roll angle, pitch angle, or yaw angle of the device with the geomagnetic field as the absolute reference system and the angle information of the magnetic field direction through a 9-axis AHRS system (Attitude and Heading Reference System). Exemplarily, a 9-axis AHRS algorithm can be applied. Among them, the AHRS system includes a three-axis gyroscope, an accelerometer, and a magnetometer. The 9-axis AHRS algorithm can be used to obtain the pose information in the geomagnetic field coordinate system, and then use the pose information to add constraints on scene judgment based on similarity judgment. Refer to Figure 4 , a schematic diagram of the pose of a SLAM system is shown. In Figure 4 , if it corresponds to the pose of the SLAM system in the first device, the visual camera is the visual camera mounted in the first device; if it corresponds to the pose of the SLAM system in the second device, the visual camera is the visual camera mounted in the second device.

[0065] In a specific example, the pose information of the first device at the first moment is, for example, the three-axis initial magnetic components of the magnetometer of the first device at the first moment in the geomagnetic field coordinate system, the roll angle and pitch angle of the first device at the first moment.

[0066] Taking the first device as an example, the three-axis initial magnetic components of the magnetometer of the first device at the first moment in the geomagnetic field coordinate system are [Mb x1 , Mb y1 , Mb z1 , the roll angle of the first device at the first moment is roll1 and the pitch angle is pitch1. Then, the target horizontal axis magnetic component and target vertical axis magnetic component of the first device in the geomagnetic field coordinate system at the first moment:

[0067] X h1 =M bx1 ×cos(pitch1)+M by1 ×sin(roll1)×sin(pitch1)+M bz1

[0068] ×cos(roll1)×sin(pitch1)

[0069] Y h1 =M by1 ×cos(roll1)-M bz1 ×sin(roll1)

[0070] Among them, X h1 is the target horizontal axis magnetic component of the first device in the geomagnetic field coordinate system at the first moment, and Y h1 is the target vertical axis magnetic component of the first device in the geomagnetic field coordinate system at the first moment.

[0071] A2. Determine the first magnetic heading angle of the first device in the geomagnetic coordinate system at the first moment according to the target horizontal axis magnetic component and the target vertical axis magnetic component of the first device at the first moment.

[0072] In a specific example, the first magnetic heading angle of the first device in the geomagnetic coordinate system at the first moment is determined in the following manner:

[0073]

[0074] where, φ M01 is the first magnetic heading angle of the first device in the geomagnetic coordinate system at the first moment, X h1 is the target horizontal axis magnetic component of the first device in the geomagnetic coordinate system at the first moment, and Y h1 is the target horizontal axis magnetic component of the first device in the geomagnetic coordinate system at the first moment.

[0075] For step B, after determining the first magnetic heading angle of the first device in the geomagnetic coordinate system at the first moment, apply this first magnetic heading angle and the magnetic declination of the area where the first device is located to determine the first heading angle of the first device at the first moment.

[0076] Specifically, the magnetic declinations of different regions are different, and can be specifically obtained by querying according to the geomagnetic field model WMM2005 and the position where the carrier (the first device or the second device) is located. Therefore, the magnetic declination φ 01 of the area where the first device is located can be determined, and then subtract φ M01 from the first magnetic heading angle φ 01 of the first device in the geomagnetic coordinate system at the first moment, then the first heading angle φ1 = φ M01 - φ 01 can be obtained.

[0077] Similarly, the determination process of the second heading angle of the second device at the second moment is as follows:

[0078] C. Determine the second magnetic heading angle of the second device in the geomagnetic coordinate system at the second moment according to the pose information of the second device at the second moment.

[0079] D. Apply the second magnetic heading angle and the magnetic declination of the area where the second device is located to determine the second heading angle of the second device at the second moment.

[0080] Among them, the determination process of the second magnetic heading angle mainly includes the following two steps C1 and C2:

[0081] C1. Determine the target horizontal axis magnetic component and the target vertical axis magnetic component of the second device in the geomagnetic coordinate system at the second moment according to the three-axis initial magnetic components of the magnetometer of the second device in the geomagnetic coordinate system at the second moment, and the roll angle and pitch angle of the second device at the second moment.

[0082] C2. Determine the second magnetic heading angle of the second device in the geomagnetic coordinate system at the second moment according to the target horizontal axis magnetic component and the target vertical axis magnetic component of the second device in the geomagnetic coordinate system at the second moment.

[0083] In this way, the second heading angle φ2 of the second device at the second moment can be obtained as φ2 = φ M02 - φ 02 . Where φ M02 is the second magnetic heading angle of the second device in the geomagnetic coordinate system at the second moment, and φ 02 is the magnetic declination of the area where the second device is located.

[0084] It should be noted that the determination process of the second heading angle of the second device at the second moment is the same as that of the first heading angle of the first device at the first moment, which will not be elaborated here.

[0085] Regarding S304, the above process determines the first heading angle φ1 of the first device at the first moment and the second heading angle φ2 of the second device at the second moment. If the absolute value of the difference between the two, △ = |φ1 - φ2|, is less than the preset angle threshold (such as 5°), it can be determined that the poses of the first device and the second device are the same. At this time, the maps corresponding to the first device and the second device can be fused based on the pose information of the first device and the second device respectively. For the specific fusion process, reference can be made to the map fusion method in the related technology, which will not be elaborated here.

[0086] It should be noted that taking the server as an example for the map fusion device, the map fusion function in the server is always turned on, receiving the first image collected by the first device and the second image collected by the second device in real time, and performing corresponding similarity and heading angle calculations. When the heading angle meets the fusion condition, the maps corresponding to the first device and the second device are fused. To make the technical solution of this application more perfect, the following will be described with a complete flowchart, see Figure 5 .

[0087] S501. For any first image collected by the first device, match it with at least one second image collected by the second device respectively to determine at least one similarity.

[0088] S502. If there is a target similarity greater than the preset similarity threshold among the at least one similarity, determine the target first image and the target second image corresponding to the target similarity.

[0089] S5031. Determine the target horizontal axis magnetic component and the target vertical axis magnetic component of the first device at the first moment in the geomagnetic coordinate system based on the three-axis initial magnetic components of the magnetometer of the first device at the first moment, the roll angle, and the pitch angle of the first device at the first moment.

[0090] S5032. Determine the first magnetic heading angle of the first device at the first moment in the geomagnetic coordinate system based on the target horizontal axis magnetic component and the target vertical axis magnetic component of the first device at the first moment in the geomagnetic coordinate system.

[0091] S5033. Determine the first heading angle of the first device at the first moment by applying the first magnetic heading angle and the magnetic declination of the area where the first device is located.

[0092] Wherein, the first moment is the acquisition moment of the target first image.

[0093] S5041. Determine the target horizontal axis magnetic component and the target vertical axis magnetic component of the second device at the second moment in the geomagnetic coordinate system based on the three-axis initial magnetic components of the magnetometer of the second device at the second moment, the roll angle, and the pitch angle of the second device at the second moment.

[0094] S5042. Determine the second magnetic heading angle of the second device at the second moment in the geomagnetic coordinate system based on the target horizontal axis magnetic component and the target vertical axis magnetic component of the second device at the second moment in the geomagnetic coordinate system.

[0095] S5043. Determine the second heading angle of the second device at the second moment by applying the second magnetic heading angle and the magnetic declination of the area where the second device is located.

[0096] Wherein, the second moment is the acquisition moment of the target second image.

[0097] S505. If the absolute value of the difference between the first heading angle and the second heading angle is less than the preset angle threshold, it is determined that the poses of the first device and the second device are the same, and the maps corresponding to the first device and the second device are fused based on the pose information of the first device and the second device respectively.

[0098] S506. Send the fused map to the first device and the second device respectively, so that the first device and the second device update the maps corresponding to them respectively.

[0099] In the above complete process, S5031, S5032, and S5033 are the determination processes of the first heading angle of the first device at the first moment, and S5041, S5042, and S5043 are the determination processes of the second heading angle of the second device at the second moment.

[0100] In the above embodiments, pose information is added on the premise that the image information is similar, which can largely prevent misjudgment in similar scenarios in different spaces.

[0101] As Figure 6 shown, based on the same inventive concept as the above image-based map fusion method, an embodiment of the present application further provides an image-based map fusion device, including a similarity determination module 61, a target image determination module 62, a heading angle determination module 63, and a fusion module 64.

[0102] Among them, the similarity determination module 61 is configured to match any first image collected by the first device with at least one second image collected by the second device respectively, and determine at least one similarity;

[0103] The target image determination module 62 is configured to determine a target first image and a target second image corresponding to the target similarity when there is a target similarity greater than a preset similarity threshold among the at least one similarity;

[0104] The heading angle determination module 63 is configured to determine a first heading angle of the first device at the first moment according to the pose information of the first device at the first moment, and determine a second heading angle of the second device at the second moment according to the pose information of the second device at the second moment; where the first moment is the acquisition moment of the target first image, and the second moment is the acquisition moment of the target second image;

[0105] The fusion module 64 is configured to determine that the poses of the first device and the second device are the same when the absolute value of the difference between the first heading angle and the second heading angle is less than a preset angle threshold, and fuse the maps corresponding to the first device and the second device respectively based on the pose information of the first device and the second device.

[0106] In some exemplary embodiments, the heading angle determination module 63 is specifically configured to:

[0107] Determine a first magnetic heading angle of the first device at the first moment in the geomagnetic coordinate system according to the pose information of the first device at the first moment;

[0108] Apply the first magnetic heading angle and the magnetic declination of the area where the first device is located to determine the first heading angle of the first device at the first moment;

[0109] The heading angle determination module 63 is specifically further configured to:

[0110] Determine a second magnetic heading angle of the second device at the second moment in the geomagnetic coordinate system according to the pose information of the second device at the second moment;

[0111] Apply the second magnetic heading angle and the magnetic declination of the area where the second device is located to determine the second heading angle of the second device at the second moment.

[0112] In some exemplary embodiments, the heading angle determination module 63 is further specifically configured to:

[0113] Determine the target horizontal axis magnetic component and the target vertical axis magnetic component of the first device in the geomagnetic coordinate system at the first moment according to the three-axis initial magnetic components of the magnetometer of the first device at the first moment, the roll angle and the pitch angle of the first device at the first moment;

[0114] Determine the first magnetic heading angle of the first device in the geomagnetic coordinate system at the first moment according to the target horizontal axis magnetic component and the target vertical axis magnetic component of the first device in the geomagnetic coordinate system at the first moment;

[0115] The heading angle determination module 63 is further specifically configured to:

[0116] Determine the target horizontal axis magnetic component and the target vertical axis magnetic component of the second device in the geomagnetic coordinate system at the second moment according to the three-axis initial magnetic components of the magnetometer of the second device at the second moment, the roll angle and the pitch angle of the second device at the second moment;

[0117] Determine the second magnetic heading angle of the second device in the geomagnetic coordinate system at the second moment according to the target horizontal axis magnetic component and the target vertical axis magnetic component of the second device in the geomagnetic coordinate system at the second moment.

[0118] In some exemplary embodiments, the heading angle determination module 63 is further specifically configured to determine the magnetic heading angle in the geomagnetic coordinate system by the following method:

[0119]

[0120] Wherein, The magnetic heading angle in the geomagnetic coordinate system, X h Is the target horizontal axis magnetic component of the geomagnetic coordinate system, Y h Is the target horizontal axis magnetic component of the geomagnetic coordinate system.

[0121] In some exemplary embodiments, the similarity determination module 61 is specifically configured to:

[0122] For any first image collected by the first device, extract the image feature points of the first image and integrate the feature points of the first image into a first bag-of-words vector; and, for any second image collected by the second device, extract the image feature points of the second image and integrate the feature points of the second image into a second bag-of-words vector;

[0123] Determine the similarity between the first image and the second image according to the distance between the first bag-of-words vector and the second bag-of-words vector.

[0124] In some exemplary embodiments, it further includes a sending module, which is configured to, after fusing the maps corresponding to the first device and the second device respectively based on the pose information of the first device and the second device:

[0125] Send the fused map to the first device and the second device respectively, so that the first device and the second device update the maps corresponding to them respectively.

[0126] The image-based map fusion device provided in the embodiments of the present application and the above-mentioned image-based map fusion method adopt the same inventive concept and can achieve the same beneficial effects, which will not be elaborated here.

[0127] Based on the same technical concept, the embodiments of the present application provide an image-based map fusion device. Refer to Figure 7 , this map fusion device includes a processor 701, a memory 702, and at least one external communication interface 703; the above-mentioned processor 701, memory 702, and external communication interface 703 are all connected through a bus 704.

[0128] The memory 702 stores a computer program. When the processor 701 executes this computer program, it implements the aforementioned image-based map fusion method and can achieve the same technical effects.

[0129] As an embodiment, the number of processors 701 can be one or more. The processor 701 and the memory 702 can be coupled or relatively independent.

[0130] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned computer program can be stored in a computer-readable storage medium. When this computer program is executed, it executes the steps including the above method embodiments; and the aforementioned storage medium includes: removable storage devices, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks, or optical disks and other various media that can store program codes.

[0131] Based on the same technical concept, the embodiments of the present application also provide a computer-readable storage medium. This computer-readable storage medium stores computer instructions. When the above computer instructions run on a computer, the computer is made to execute the image-based map fusion method described above.

[0132] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0133] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0134] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0136] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. An image-based map fusion method, characterized in that, Including: For any first image collected by a first device, respectively match it with at least one second image collected by a second device to determine at least one similarity; If there is a target similarity greater than a preset similarity threshold among the at least one similarity, determine the target first image and the target second image corresponding to the target similarity; Determine the first heading angle of the first device at the first moment according to the pose information of the first device at the first moment, and determine the second heading angle of the second device at the second moment according to the pose information of the second device at the second moment; wherein, the first moment is the acquisition moment of the target first image, and the second moment is the acquisition moment of the target second image; If the absolute value of the difference between the first heading angle and the second heading angle is less than a preset angle threshold, determine that the poses of the first device and the second device are the same, and perform map fusion on the target first image and the target second image based on the respective pose information of the first device and the second device; Wherein, the first heading angle of the first device at the first moment is determined by applying a first magnetic heading angle and the magnetic declination of the area where the first device is located, the first magnetic heading angle is the angle of the first device in the geomagnetic coordinate system, and the first magnetic heading angle is determined according to the pose information of the first device at the first moment; The second heading angle of the second device at the second moment is determined by applying a second magnetic heading angle and the magnetic declination of the area where the second device is located, the second magnetic heading angle is the angle of the second device in the geomagnetic coordinate system, and the second magnetic heading angle is determined according to the pose information of the second device at the second moment.

2. The method according to claim 1, characterized in that The first magnetic heading angle is determined by the following method: According to the three-axis initial magnetic components of the magnetometer of the first device at the first moment obtained in the geomagnetic coordinate system, the roll angle and pitch angle of the first device at the first moment, determine the target horizontal axis magnetic component and target vertical axis magnetic component of the first device in the geomagnetic coordinate system at the first moment; Determine the first magnetic heading angle of the first device in the geomagnetic coordinate system at the first moment according to the target horizontal axis magnetic component and target vertical axis magnetic component of the first device in the geomagnetic coordinate system at the first moment; The second magnetic heading angle is determined by the following method: According to the three-axis initial magnetic components of the magnetometer of the second device at the second moment obtained in the geomagnetic coordinate system, the roll angle and pitch angle of the second device at the second moment, determine the target horizontal axis magnetic component and target vertical axis magnetic component of the second device in the geomagnetic coordinate system at the second moment; Determine the second magnetic heading angle of the second device in the geomagnetic coordinate system at the second moment according to the target horizontal axis magnetic component and target vertical axis magnetic component of the second device in the geomagnetic coordinate system at the second moment.

3. The method according to claim 2, wherein Determine the magnetic heading angle in the geomagnetic coordinate system by the following method: Among them, the magnetic heading angle in the geomagnetic coordinate system, X h is the target horizontal axis magnetic component of the geomagnetic coordinate system, Y h is the target horizontal axis magnetic component of the geomagnetic coordinate system.

4. The method according to claim 1, characterized in that For any first image collected by the first device, respectively matching it with at least one second image collected by the second device to determine at least one similarity, including: For any first image collected by the first device, extracting the image feature points of the first image and integrating the feature points of the first image into a first bag-of-words vector; and, for any second image collected by the second device, extracting the image feature points of the second image and integrating the feature points of the second image into a second bag-of-words vector; Determining the similarity between the first image and the second image according to the distance between the first bag-of-words vector and the second bag-of-words vector.

5. The method according to any one of claims 1 to 4, characterized in that After fusing the maps corresponding to the first device and the second device respectively based on the pose information of the first device and the second device, the method further includes: Sending the fused map to the first device and the second device respectively, so that the first device and the second device update the maps corresponding to them respectively.

6. An image-based map fusion device, characterized in that, Including a processor, a memory, and at least one external communication interface, where the processor, the memory, and the external communication interface are all connected through a bus; The external communication interface is configured to receive any first image collected by the first device and at least one second image collected by the second device; A computer program is stored in the memory, and the processor is configured to perform the following operations based on the computer program: For any first image collected by the first device, respectively matching it with at least one second image collected by the second device to determine at least one similarity; If there is a target similarity greater than a preset similarity threshold among the at least one similarity, determining the target first image and the target second image corresponding to the target similarity; Determining the first heading angle of the first device at the first moment according to the pose information of the first device at the first moment, and determining the second heading angle of the second device at the second moment according to the pose information of the second device at the second moment; where the first moment is the acquisition moment of the target first image, and the second moment is the acquisition moment of the target second image; If the absolute value of the difference between the first heading angle and the second heading angle is less than a preset angle threshold, determining that the poses of the first device and the second device are the same, and performing map fusion on the target first image and the target second image based on the pose information of the first device and the second device; Where the first heading angle of the first device at the first moment is determined by applying the first magnetic heading angle and the magnetic declination of the area where the first device is located, the first magnetic heading angle is the angle of the first device in the geomagnetic coordinate system, and the first magnetic heading angle is determined according to the pose information of the first device at the first moment; The second heading angle of the second device at the second moment is determined by applying the second magnetic heading angle and the magnetic declination of the area where the second device is located. The second magnetic heading angle is the angle of the second device in the geomagnetic coordinate system, and the second magnetic heading angle is determined according to the pose information of the second device at the second moment.

7. The device according to claim 6, characterized in that, The processor is further configured to: Determine the target horizontal axis magnetic component and the target vertical axis magnetic component of the first device in the geomagnetic coordinate system at the first moment according to the three-axis initial magnetic components of the magnetometer of the first device at the first moment in the geomagnetic coordinate system, the roll angle and the pitch angle of the first device at the first moment; Determine the first magnetic heading angle of the first device in the geomagnetic coordinate system at the first moment according to the target horizontal axis magnetic component and the target vertical axis magnetic component of the first device in the geomagnetic coordinate system at the first moment; Determine the target horizontal axis magnetic component and the target vertical axis magnetic component of the second device in the geomagnetic coordinate system at the second moment according to the three-axis initial magnetic components of the magnetometer of the second device at the second moment in the geomagnetic coordinate system, the roll angle and the pitch angle of the second device at the second moment; Determine the second magnetic heading angle of the second device in the geomagnetic coordinate system at the second moment according to the target horizontal axis magnetic component and the target vertical axis magnetic component of the second device in the geomagnetic coordinate system at the second moment.

8. The device according to claim 7, characterized in that, The processor is further configured to determine the magnetic heading angle in the geomagnetic coordinate system by the following method: Among them, the magnetic heading angle in the geomagnetic coordinate system, X h is the target horizontal axis magnetic component of the geomagnetic coordinate system, Y h is the target horizontal axis magnetic component of the geomagnetic coordinate system.

Images