Fusion frame acquisition method, device, SLAM system and storage medium

Abstract

The present invention discloses a method for obtaining a fused frame, the method comprising the following steps: obtaining N target images of a target area through the N cameras, N being a positive integer greater than or equal to 2; performing feature extraction on the N target images to obtain a feature point set; obtaining a hash table based on the feature point set; obtaining a map point set based on the hash table; obtaining a description subset based on the hash table; and obtaining a fused frame based on the hash table, the map point set and the description subset. The present invention also discloses a fused frame obtaining device, a SLAM system and a storage medium. The fused frame obtained by the present invention includes more visual feature information, so that the accuracy of the fused frame is better.

Description

Technical Field

[0001] The present invention relates to the field of feature information processing, and in particular to a fusion frame obtaining method, device, SLAM system and storage medium. Background Art

[0002] At present, the mainstream visual SLAM (simultaneous localization and mapping) systems include monocular cameras, stereo vision cameras or depth color cameras. The SLAM system obtains target images of the target area at different times through a monocular camera, and obtains fused frames by fusing the target images.

[0003] However, when the fused frame is obtained by the existing method, the accuracy of the fused frame is poor. Summary of the invention

[0004] The main purpose of the present invention is to provide a fusion frame acquisition method, device, SLAM system and storage medium, aiming to solve the technical problem of poor accuracy of fusion frames in the prior art.

[0005] To achieve the above object, the present invention proposes a fusion frame acquisition method, which is applied to a SLAM system including N cameras, and the method comprises the following steps:

[0006] Acquire N target images of the target area through the N cameras, where N is a positive integer greater than or equal to 2;

[0007] Performing feature extraction on the N target images to obtain a feature point set;

[0008] Based on the feature point set, a hash table is obtained;

[0009] Based on the hash table, obtaining a map point set;

[0010] Based on the hash table, obtaining a description subset;

[0011] A fused frame is obtained based on the hash table, the map point set and the description subset.

[0012] Optionally, the step of obtaining a hash table based on the feature point set includes:

[0013] Determining matching feature points that meet preset matching conditions in the feature point set;

[0014] Concentrating the feature points to determine the monocular feature points that do not meet the preset matching condition;

[0015] A hash table is obtained based on the matching feature points and the monocular feature points.

[0016] Optionally, the step of obtaining a hash table based on the matching feature points and the monocular feature points includes:

[0017] Obtaining a first descriptor, a first pixel coordinate, and a first camera ID of the matching feature point in the N target images;

[0018] Obtain a second descriptor, a second pixel coordinate, and a second camera ID of the monocular feature point in the N target images;

[0019] A hash table is obtained based on the matching feature points, the monocular feature points, the first descriptor, the first pixel coordinates, the first camera ID, the second descriptor, the second pixel coordinates and the second camera ID.

[0020] Optionally, the step of obtaining a map point set based on the hash table includes:

[0021] Based on the hash table, obtaining the selected pixel coordinates corresponding to the matching feature points;

[0022] Based on the selected pixel coordinates of the matching feature points, obtaining the selected camera coordinates of the matching feature points;

[0023] Based on the selected camera coordinates of the matching feature points, obtaining depth information of the matching feature points;

[0024] Based on the depth information of the matching feature points, obtaining the world coordinates of the matching feature points;

[0025] The map point set is obtained based on the world coordinates of the matching feature points and the matching feature points.

[0026] Optionally, the step of obtaining the depth information of the matching feature point based on the selected camera coordinates of the matching feature point includes:

[0027] determining two result camera coordinates in the selected camera coordinates of the matching feature points;

[0028] According to the two result camera coordinates, the depth information of the matching feature points is obtained by triangulation and least square method.

[0029] Optionally, the step of obtaining the world coordinates of the matching feature points based on the depth information of the matching feature points includes:

[0030] The world coordinates of the matching feature point are obtained based on the depth information of the matching feature point, the target camera coordinates corresponding to the depth information, the rotation matrix of the target camera coordinate system and the world coordinate system corresponding to the target camera coordinates, and the translation vector between the target camera coordinate system and the world coordinate system.

[0031] Optionally, the step of obtaining a description subset based on the hash table includes:

[0032] Based on the hash table, obtaining a first selected descriptor corresponding to the matching feature point and a second selected descriptor corresponding to the monocular feature point;

[0033] Determining, in the first selected descriptor, a first result descriptor when the matching feature point is first extracted;

[0034] Determining, in the second selected descriptor, a second result descriptor when the monocular feature point is extracted for the first time;

[0035] The descriptor set is obtained according to the matching feature points, the monocular feature points, the first result descriptor and the second result descriptor.

[0036] In addition, to achieve the above purpose, the present invention also proposes a fusion frame acquisition device, which is applied to a SLAM system including N cameras, and the device includes:

[0037] An acquisition module, used for acquiring N target images of a target area, where N is a positive integer greater than or equal to 2;

[0038] An extraction module, used for performing feature extraction on the N target images to obtain a feature point set;

[0039] A first obtaining module, used for obtaining a hash table based on the feature point set;

[0040] A second obtaining module, used for obtaining a map point set based on the hash table;

[0041] A third obtaining module, used for obtaining a description subset based on the hash table;

[0042] The fused frame obtaining module is used to obtain a fused frame based on the hash table, the map point set and the description subset.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also proposes a SLAM system, which includes: N cameras, a memory, a processor, and a fusion frame acquisition program stored in the memory and running on the processor, and when the fusion frame acquisition is executed by the processor, the steps of the fusion frame acquisition method described in any one of the above items are implemented; wherein N is a positive integer greater than or equal to 2.

[0044] In addition, to achieve the above purpose, the present invention also proposes a storage medium, on which a fusion frame acquisition program is stored, and when the fusion frame acquisition program is executed by a processor, the steps of any of the above-mentioned fusion frame acquisition methods are implemented. .

[0045] The technical solution of the present invention adopts a fusion frame acquisition method, which is applied to a SLAM system including N cameras. The method includes the following steps: acquiring N target images of the target area through the N cameras, N being a positive integer greater than or equal to 2; performing feature extraction on the N target images to obtain a feature point set; obtaining a hash table based on the feature point set; obtaining a map point set based on the hash table; obtaining a description subset based on the hash table; and obtaining a fusion frame based on the hash table, the map point set and the description subset. Since the multiple target images of the N cameras include more visual feature information than the target image of a monocular camera, and at the same time, the fusion frame obtained by the present invention based on the multiple target images acquired by the N cameras is not based on the target image acquired by the monocular camera. Therefore, the fusion frame obtained by the present invention includes more visual feature information, so that the accuracy of the fusion frame is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of the SLAM system structure of the hardware operating environment involved in the embodiment of the present invention;

[0048] Figure 2 It is a flowchart of a first embodiment of a method for obtaining a fused frame according to the present invention;

[0049] Figure 3 Hash representation for the present invention;

[0050] Figure 4 is a schematic diagram of the triangulation method;

[0051] Figure 5 It is a structural diagram of the fusion frame;

[0052] Figure 6 This is a structural block diagram of the first embodiment of the fusion frame obtaining device of the present invention.

[0053] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Reference Figure 1 , is a schematic diagram of the SLAM system structure of the hardware operating environment involved in the embodiment of the present invention.

[0056] Typically, the SLAM system includes: a camera 307, at least one processor 301, a memory 302, and a fusion frame acquisition program stored in the memory and executable on the processor, wherein the fusion frame acquisition program is configured to implement the steps of the fusion frame acquisition method as described above.

[0057] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. The processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process operations related to the fusion frame acquisition method, so that the fusion frame acquisition method model can be trained and learned autonomously to improve efficiency and accuracy.

[0058] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 801 to implement the fusion frame acquisition method provided in the method embodiment of the present application.

[0059] Camera 307 includes N cameras, and there may be a rotation angle and distance between the N cameras. The N cameras are used to obtain a target image of the target area. Usually, at the same time, the N cameras simultaneously obtain the target image of the target area. Due to the rotation angle and distance between the multiple cameras, the multiple target images obtained are different. Among them, the N cameras can be high-definition cameras or ordinary cameras, and the cameras in the N cameras can be monocular cameras, stereo vision cameras, and depth color cameras.

[0060] In some embodiments, the SLAM system may also optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected via a bus or a signal line. Each peripheral device may be connected to the communication interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 304, a display screen 305 and a power supply 306.

[0061] The communication interface 303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0062] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 304 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 304 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 304 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0063] The display screen 305 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 305 is a touch display screen, the display screen 305 also has the ability to collect touch signals on the surface or above the surface of the display screen 305. The touch signal can be input to the processor 301 as a control signal for processing. At this time, the display screen 305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 305 can be one, the front panel of the electronic device; in other embodiments, the display screen 305 can be at least two, respectively arranged on different surfaces of the electronic device or in a folding design; in some other embodiments, the display screen 305 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device. Even, the display screen 305 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 305 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode) and the like.

[0064] The power supply 306 is used to power various components in the electronic device. The power supply 306 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 306 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology. Those skilled in the art will understand that Figure 1The structure shown in the figure does not constitute a limitation on the fused frame acquisition device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0065] In addition, an embodiment of the present invention further proposes a storage medium, on which a fusion frame acquisition program is stored, and when the fusion frame acquisition program is executed by a processor, the steps of the fusion frame acquisition method as described above are implemented. Therefore, no further description will be given here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in the present application, please refer to the description of the method embodiment of the present application. Determined as an example, the program instructions can be deployed to execute on a SLAM system, or on multiple SLAM systems located at one location, or, on multiple SLAM systems distributed at multiple locations and interconnected by a communication network.

[0066] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0067] Based on the above hardware structure, an embodiment of the method for obtaining a fused frame of the present invention is proposed.

[0068] Reference Figure 2 , Figure 2 The present invention is a flowchart of a first embodiment of a method for obtaining a fusion frame, wherein the method is applied to a SLAM system including multiple cameras, and the method includes the following steps:

[0069] Step S11: Acquire N target images of the target area through the N cameras, where N is a positive integer greater than or equal to 2.

[0070] It should be noted that the N target images refer to N target images at the same time. The SLAM system includes N cameras. At the same time, the N cameras simultaneously capture N target images of the target area, and one camera captures one target image; step S11 is executed at each moment, and multiple fused frames can be obtained by executing step S11 multiple times. It can be understood that the method of the present invention is a process for obtaining a fused frame, and the process for obtaining multiple fused frames is to execute the method of the present invention multiple times.

[0071] For example, N cameras are 4 cameras, and target images of the target area captured at the same time are 4 target images.

[0072] In specific applications, N cameras may have a continuous shooting function, and the target images obtained by the N cameras at the same moment are multiple images with a time correspondence in the continuous shooting images taken by multiple cameras; when the N cameras shoot video feature information, the multiple target images obtained by the N cameras at the same moment are multiple video frames with a time correspondence in the video information.

[0073] The target area may refer to the area that the SLAM system passes through when it is running, and the characteristic information included in the target area can be captured by the multiple cameras of the SLAM system. The target area may be an unknown area or a known area, and the present invention does not limit this.

[0074] Step S12: extracting features from the N target images to obtain a feature point set.

[0075] It should be noted that in image processing, feature points refer to points where the grayscale value of an image changes dramatically or points with large curvature on the edge of an image. Features can be divided into color features and texture features. Among them, color features are a global feature that describes the surface properties of the scene corresponding to an image or an image region. Generally, color features are based on pixel features. At this time, all pixels belonging to an image or an image region have their own contributions. Since color is insensitive to changes in the direction, size, etc. of an image or an image region, color features cannot capture the local features of objects in an image well. In addition, when only color features are used for querying, if the database is large, many unnecessary images will often be retrieved. Color histogram is the most commonly used method to express color features. Its advantage is that it is not affected by image rotation and translation changes. Further, with the help of normalization, it is not affected by changes in image scale. The basic disadvantage is that it does not express the characteristic information of color space distribution. Texture features are also a global feature that also describes the surface properties of the scene corresponding to an image or an image region.

[0076] In a specific application, the feature points may be feature points extracted by any of the orb, surf, or sift feature extraction algorithms, or feature points extracted by other methods, and the present invention does not limit this.

[0077] Among them, the feature point includes the camera ID, descriptor and pixel coordinates (2D coordinates) of the feature point; the descriptor can be a vector describing the pixel information around the feature point, and the vector can be a binary number. The camera ID refers to the number of the feature point in the target image. Since the same feature point may be extracted in N target images at the same time, the same feature point can correspond to N camera IDs at the same time; the descriptor refers to the descriptor of the feature point in the target image. The descriptors of the same feature point in different target images at the same time may be different. The pixel coordinates refer to the pixel coordinates of the feature point in the target image. The pixel coordinates of the same feature point in different target images at the same time may be different.

[0078] It can be understood that, for ease of understanding, the target points mentioned below all refer to objective physical points corresponding to the feature points.

[0079] For example, the SLAM system has four cameras, camera a, camera b, camera c and camera d. At the same time, the four cameras capture four target images of the target area, perform feature extraction on the four target images, and extract feature points A corresponding to the target area. In the target image of camera a, the feature point camera ID is 12, the feature point A descriptor is 1010100, and the pixel coordinates are (13, 11). In the target image of camera b, the feature point camera ID is 22, the feature point A descriptor is 1010110, and the pixel coordinates are (23, 11). In the target image of camera c, the feature point camera ID is 32, the feature point A descriptor is 1010111, and the pixel coordinates are (33, 11). In the target image of camera d, the feature point camera ID is 32, the feature point A descriptor is 1010101, and the pixel coordinates are (43, 11). At this time, there are 4 descriptors corresponding to feature point A, 4 camera IDs, and 4 pixel coordinates.

[0080] According to the extracted feature points, a feature point set including the feature points and feature information of the feature points is obtained, where the feature information of the feature points is the camera ID, pixel coordinates and descriptors.

[0081] Step S13: Obtain a hash table based on the feature point set.

[0082] Further, step S13 includes: determining matching feature points that meet preset matching conditions from the feature point set; determining monocular feature points that do not meet the preset matching conditions from the feature point set; and obtaining a hash table based on the matching feature points and the monocular feature points.

[0083] It should be noted that when the descriptors of multiple feature points are similar, the multiple feature points meet the preset matching conditions. The present invention does not limit the degree of similarity of the descriptors, and users can set it according to their own needs. However, in order to ensure the high accuracy of the obtained matching feature points, it is better to set the degree of similarity of the descriptors of the preset matching conditions to be higher; for example, the descriptor is a 7-bit binary number, and the preset matching condition is that the first five bits of the descriptor are the same.

[0084] Furthermore, the step of obtaining a hash table based on the matching feature points and the monocular feature points includes: obtaining a first descriptor, a first pixel coordinate, and a first camera ID of the matching feature points in the N target images; obtaining a second descriptor, a second pixel coordinate, and a second camera ID of the monocular feature points in the N target images; and obtaining a hash table based on the matching feature points, the monocular feature points, the first descriptor, the first pixel coordinate, the first camera ID, the second descriptor, the second pixel coordinate, and the second camera ID.

[0085] It should be noted that after the feature points are extracted, the feature points extracted from the target images corresponding to different cameras at the same time are taken as a set, that is, N cameras correspond to N sets. All feature points in the N sets are matched, and the feature points that meet the preset matching conditions are combined into a matching feature point. The matching feature point corresponds to multiple camera IDs, multiple descriptors, and multiple pixel coordinates as the feature information of the matching feature point, wherein not all target images of each camera can extract matching feature points, but only the target images of at least two cameras can extract matching feature points, so the multiple here refers to a value not greater than N; the feature points in the N sets that do not meet the preset matching conditions are taken as monocular feature points, and the feature information of the monocular feature points is a camera ID, a descriptor, and a pixel coordinate corresponding to the feature point.

[0086] For example, the SLAM system has four cameras, camera a, camera b, camera c and camera d. At the same time, the four cameras capture four target images of the target area, and feature extraction is performed on the four target images. The feature point A corresponding to the target point in the target area is extracted. In the target image of camera a, the camera ID of feature point A is 12, the descriptor is 1010100, and the pixel coordinates are (13, 11). In the target image of camera b, the camera ID of feature point A is 22, the descriptor is 1010110, and the pixel coordinates are (23, 11). In the target image of camera c, the camera ID of feature point A is 32, the descriptor is 101011 1, the pixel coordinates are (33, 11). In the target image of camera d, the camera ID of feature point A is 32, the descriptor is 1010101, and the pixel coordinates are (43, 11). At this time, feature point A is a matching feature point. There are 4 corresponding descriptors, 4 camera IDs, and 4 pixel coordinates. The feature information of matching feature point A includes four camera IDs - 12, 22, 32 and 42, 4 descriptors - 1010100, 1010110, 1010111 and 1010101, and 4 pixel coordinates - (13, 11), (23, 11), (33, 11) and (43, 11).

[0087] Reference Figure 3 , Figure 3 For the hash table scheme of the present invention, all matching feature points and monocular feature points are assigned values ​​of 0 to (M-1), that is, the sum of matching feature points and monocular feature points is M, and the key value id (key) of the hash table is assigned, one matching feature point or one monocular feature point corresponds to one key value, and the feature information of the matching feature point and the monocular feature point is used as the value of the hash table. Wherein, the SLAM system includes N cameras, so the value of a key value in the hash table includes N different feature information, that is, a target point corresponds to a feature information of a feature point in N target images of N cameras, and the value of the hash table is the camera ID, and the descriptor and pixel coordinates are attached to the corresponding camera ID in the mode of sub-feature information; wherein, the value of the hash table can be the camera ID attached to the descriptor and pixel coordinates, or it can be the information obtained by combining the descriptor, pixel coordinates and camera ID, and the present invention does not limit it.

[0088] It can be understood that the first value of a key value (i.e., a feature point corresponding to a target point, which can be a monocular feature point or a matching feature point) is the corresponding camera ID when the feature point is extracted from the target image of the first camera, and the camera corresponding to the value can be determined by the position of the value; in addition, when the feature point cannot be extracted in a certain target image, the camera ID of the feature point in the camera corresponding to the target image is assigned -1, and the camera ID is not attached to the descriptor and pixel coordinates, wherein the camera ID is assigned -1, which means that the feature point is not extracted in the target image of the camera, and the assignment can also be in other forms, such as -2, and the present invention is not limited to -1.

[0089] exist Figure 3 There are N feature information of the feature point corresponding to the key value 104, which are, from left to right, not extracted in the target image of camera 1...extracted in the target image of camera i, and the corresponding camera ID is 201; extracted in the target image of camera N, and the corresponding camera ID is 3.

[0090] It is understandable that the hash table diagram provided by the present invention is a preferred choice, and other types of hash tables can also be determined based on the ideas of the present invention, which will not be described in detail here.

[0091] Step S14: Based on the hash table, a map point set is obtained.

[0092] Further, step S14 includes: based on the hash table, obtaining the selected pixel coordinates corresponding to the matching feature points; based on the selected pixel coordinates of the matching feature points, obtaining the selected camera coordinates of the matching feature points; based on the selected camera coordinates of the matching feature points, obtaining the depth information of the matching feature points; based on the depth information of the matching feature points, obtaining the world coordinates of the matching feature points; based on the world coordinates of the matching feature points and the matching feature points, obtaining the map point set.

[0093] It should be noted that, referring to the structure of the above hash table, through the hash table key value, that is, the assignment of the feature point, the feature information of the matching feature point corresponding to the key value is found, and the camera to which the feature information belongs is determined as the selected camera, and the camera coordinate system to which the selected camera belongs is the selected camera coordinate system; it can be understood that when a value Q of a key value P in the hash table is -1, the camera to which the value Q corresponding to the key value P belongs is not the selected camera; the pixel coordinates of the matching feature point in the target image of the selected camera are the selected pixel coordinates, and the corresponding normalized selected camera coordinates are obtained using the selected pixel coordinates. Based on the selected camera coordinates of the matching feature point, the depth information of the matching feature point is obtained; based on the depth information of the matching feature point, the world coordinates of the matching feature point are obtained; based on the world coordinates of the matching feature point and the matching feature point, the map point set is obtained.

[0094] Among them, the depth information refers to the depth information of the matching feature point in the camera coordinate system to which the selected camera belongs. It can be understood that only one selected camera coordinate and a corresponding depth information are needed to obtain the world coordinates of the corresponding matching feature point; that is, for all matching feature points, it is sufficient to obtain one depth information for each matching feature point, and there is no need to obtain the depth information of all the selected pixel coordinate systems of the matching feature point.

[0095] For example, there are 4 cameras, and feature point A is extracted from the target images of 3 cameras among the 4 cameras, that is, feature point A is a matching feature point. In the obtained hash table, the key value of matching feature point A is 15, and there are 4 corresponding values, one of which is -1, that is, the matching feature point A is not extracted from the target image of the camera to which the -1 value belongs; then, the cameras to which the other 3 values ​​belong are selected cameras, and the pixel coordinates corresponding to the matching feature point A in the selected camera are the selected pixel coordinates, and the camera coordinates corresponding to the selected pixel coordinates are the selected camera coordinates. Among them, the 3 selected camera coordinates are H, J, and L. When the world coordinates are obtained using the 3 camera coordinates, it is only necessary to use the depth information (l) of any selected camera coordinate (for example, L) among the 3 selected camera coordinates, and the world coordinates of feature point A can be obtained using l and L.

[0096] In addition, the corresponding selected camera coordinates are obtained through formula 1 and the selected pixel coordinates.

[0097] The formula 1 is:

[0098]

[0099]

[0100] Among them, u is the horizontal coordinate in the selected pixel coordinates, v is the vertical coordinate in the selected pixel coordinates, is the horizontal coordinate of the selected camera coordinates, is the ordinate of the selected camera coordinates, f x , f y , c x , c y are all the internal parameters of the selected camera; it is understandable that the internal parameters of the same feature point corresponding to different cameras may be different; the coordinates obtained according to formula 1 only include the horizontal coordinate and the vertical coordinate Therefore, the camera coordinates at this time are normalized camera coordinates, expressed as

[0101] Furthermore, the step of obtaining the depth information of the matching feature points based on the selected camera coordinates of the matching feature points includes: determining two result camera coordinates in the selected camera coordinates of the matching feature points; and obtaining the depth information of the matching feature points by triangulation and least squares method based on the two result camera coordinates.

[0102] It can be understood that there are at least two feature information corresponding to the matching feature points, namely Figure 3 In the hash table shown, there are at least two non-1 values ​​corresponding to the matching feature points; when the number of matching feature points is large, that is, when the key values ​​corresponding to the matching feature points in the hash table are large in number and non-1, feature information corresponding to any two non-1 values ​​is determined as result feature information, and the camera coordinates obtained using the pixel coordinates in the result feature information are used as result camera coordinates, and the camera to which the result camera coordinates of the matching feature points belong is the result camera.

[0103] It should be noted that the formula of the triangulation method is as follows, that is, Formula 2:

[0104] s2X2=s1RX1+t

[0105] Among them, X1 is one of the two resulting camera coordinates, X2 is the other of the two resulting camera coordinates, s1 is the depth information of X1 in the corresponding camera coordinate system, s2 is the depth information of X2 in the corresponding camera coordinate system, R is the rotation matrix of the camera coordinate system corresponding to X1 and the camera coordinate system corresponding to X2, and t is the translation vector of the camera coordinate system corresponding to X1 and the camera coordinate system corresponding to X2. It can be understood that the camera coordinate X can be expressed as above

[0106] Reference Figure 4 , Figure 4It is a schematic diagram of the triangulation method, P is the original position map corresponding to a feature point in the target image of the SLAM system, that is, the actual position of the feature point in the world coordinates, I1 is the target image in the result camera, P1 is the coordinate pixel of P in I1, I2 is the target image in another result camera, P2 is the coordinate pixel of P in I2.

[0107] In a specific application, the depth information of any camera coordinate in the two result camera coordinates is obtained by using formula 2 and the least square method. The result camera coordinate corresponding to the depth information is the target camera coordinate. For example, when s1 in formula 2 is used as the depth information, X1 corresponding to the depth information is the target camera coordinate. The original camera coordinate corresponding to the target camera coordinate is restored by using the depth information, that is, the target camera coordinate The horizontal coordinate, vertical coordinate and depth in are multiplied by the depth information Z, respectively, where the original camera coordinates are the coordinates in the camera coordinate system to which the target camera coordinates belong. For example, X1 is the result camera coordinate of feature point A of camera 1, and the depth information s1 of X1 is used as the selected depth information to obtain the original camera coordinates (X, Y, Z) of X1, which are the coordinates in the camera coordinate system to which camera 1 belongs.

[0108] Furthermore, the step of obtaining the world coordinates of the matching feature points based on the depth information of the matching feature points includes: obtaining the world coordinates of the matching feature points based on the depth information of the matching feature points, the target camera coordinates corresponding to the depth information, the rotation matrix of the target camera coordinate system and the world coordinate system corresponding to the target camera coordinates, and the translation vector between the target camera coordinate system and the world coordinate system.

[0109] It can be understood that for each matching feature point, only one depth information needs to be obtained, and the original camera coordinates corresponding to the feature point are obtained based on the depth information and the target camera coordinates corresponding to the depth information. Among them, the camera coordinate system to which the original camera belongs obtained based on the target camera coordinates is the target camera coordinate system. For example, X1 is one of the result camera coordinates of feature point A of camera No. 1, and the depth information s1 of X1 is selected as the depth information (at this time, the result camera coordinate X1 is the target camera coordinate), and the original camera coordinates (X, Y, Z) of X1 are obtained. The original camera coordinates are the coordinates in the camera coordinate system to which camera No. 1 belongs, and the camera coordinate system to which camera No. 1 belongs is the target camera coordinate system. Then, according to the rotation matrix of the target camera coordinate system and the world coordinate system and the translation vector of the target camera coordinate system and the world coordinate system, the world coordinates of the matching feature point are obtained, and the world coordinates of the original camera coordinates are obtained. The world coordinates of the original camera coordinates are the world coordinates of the feature point.

[0110] In a specific application, the leftmost camera among the N cameras in the SLAM system can be used as camera No. 1, and the coordinate system of camera No. 1 at the initial moment can be used as the world coordinate system. The world coordinate system can also be determined in other ways, which is not limited by the present invention.

[0111] When using the above-mentioned hash table to obtain the world coordinates of the matching feature points, the key value of the hash table is used as the map point element of the map point set, the world coordinates of the matching feature points corresponding to the key value are used as the world coordinates of the key value, and the world coordinates of the monocular feature points corresponding to the key value are assigned to 0. According to the key value and the world coordinates, the map point set is obtained. It can be understood that if the number of key values ​​is M, the number of map points included in the map point set is also M, wherein the world coordinates of the map points corresponding to the monocular feature points are 0, and the world coordinates of the map points corresponding to the matching feature points are not 0 and are meaningful, wherein the map points are defined as matching feature points or monocular feature points with world coordinates.

[0112] Step S15: Based on the hash table, obtain a description subset.

[0113] Further, step S15 includes: based on the hash table, obtaining a first selected descriptor corresponding to the matching feature point and a second selected descriptor corresponding to the monocular feature point; determining a first result descriptor when the matching feature point is first extracted in the first selected descriptor; determining a second result descriptor when the monocular feature point is first extracted in the second selected descriptor; and obtaining the descriptor set according to the matching feature point, the monocular feature point, the first result descriptor and the second result descriptor.

[0114] It should be noted that the key value of each feature point in the hash table corresponds to multiple values, and each value includes feature information (descriptor, camera ID and pixel coordinates) of the feature point under different cameras. For the matching feature points, the key value corresponds to multiple feature information, and the key value of the monocular feature point corresponds to one feature information; in the matching feature points, the valid feature information (that is, the feature information corresponding to the camera ID whose value is not -1) is determined as the selected feature information, and in the selected feature information, the descriptor in the target image corresponding to the first time the matching feature point is extracted is determined as the first result descriptor; in the monocular feature points, the only descriptor is determined as the second result descriptor. It can be understood that the descriptor must be the descriptor corresponding to the first extraction of the feature point.

[0115] The key value in the hash table is used as the element unit of the description subset, and the corresponding first result descriptor or second result descriptor is used as the selected descriptor of the key value. The description subset is obtained according to the key value, the first result descriptor and the second result descriptor. It can be understood that if the number of key values ​​is M, the number of selected descriptors is also M, wherein the sum of the matching feature points and monocular feature points corresponding to the M key values ​​is M.

[0116] Step S16: Obtain a fused frame based on the hash table, the map point set and the description subset.

[0117] Reference Figure 4 , Figure 4 is a structural diagram of a fusion frame, wherein the fusion frame is obtained based on the target images of N cameras at a certain moment, and the shapes of the frames (i.e., target images) corresponding to the cameras (representing the angles and positions of the target images) are different, wherein the pose refers to the different angles and displacements, and the virtual fusion frame is the fusion frame, including a map point set, a hash table, and a description subset, wherein the sum of matching feature points and monocular feature points is M, the number of map points in the corresponding map point set is M, and the number of descriptors in the description subset is also M; the map point set originates from the map points, schematically, the map points are 5 map points - P1, P2, P3, P4 and P5, and the map points originate from the feature points, schematically, the feature points include 9 points, and there are matching feature points and monocular feature points among the 9 points.

[0118] The technical solution of this embodiment adopts a fusion frame acquisition method, which is applied to a SLAM system including N cameras. The method includes the following steps: acquiring N target images of the target area through the N cameras, where N is a positive integer greater than or equal to 2; performing feature extraction on the N target images to obtain a feature point set; obtaining a hash table based on the feature point set; obtaining a map point set based on the hash table; obtaining a description subset based on the hash table; and obtaining a fusion frame based on the hash table, the map point set and the description subset. Since the multiple target images of the N cameras include more visual feature information than the target image of the monocular camera, and at the same time, the fusion frame obtained by the present invention based on the multiple target images acquired by the N cameras is not based on the target image acquired by the monocular camera. Therefore, the fusion frame obtained by the present invention includes more visual feature information, so that the accuracy of the fusion frame is better.

[0119] At the same time, when the hash table of the above structure is used to obtain the description subset and the map point set, the key value corresponding to the feature point can be used to quickly search for the feature information of each feature point, and the search process is simple and the search efficiency is high.

[0120] Reference Figure 5 , Figure 5This is a structural block diagram of a first embodiment of a fusion frame acquisition device of the present invention, which is applied to a SLAM system including N cameras. The device includes:

[0121] An acquisition module 10 is used to acquire N target images of a target area, where N is a positive integer greater than or equal to 2;

[0122] An extraction module 20, configured to extract features from the N target images to obtain a feature point set;

[0123] A first obtaining module 30, configured to obtain a hash table based on the feature point set;

[0124] A second obtaining module 40 is used to obtain a map point set based on the hash table;

[0125] A third obtaining module 50 is used to obtain a description subset based on the hash table;

[0126] The fused frame obtaining module 60 is used to obtain a fused frame based on the hash table, the map point set and the description subset.

[0127] The above descriptions are only optional embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings, or directly / indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A method for obtaining a fused frame, characterized in that: Applied to a SLAM system including N cameras, the method comprises the following steps: Acquire N target images of the target area through the N cameras, where N is a positive integer greater than or equal to 2; Performing feature extraction on the N target images to obtain a feature point set, wherein the feature point set includes matching feature points that meet a preset matching condition and monocular feature points that do not meet the preset matching condition; Based on the feature point set, a hash table is obtained; Based on the hash table, obtaining a map point set; Based on the hash table, obtaining a description subset; Obtaining a fused frame based on the hash table, the map point set and the description subset; The step of obtaining a hash table based on the feature point set includes: Obtain a first descriptor, a first pixel coordinate, and a first camera ID of the matching feature point in the N target images; Obtain a second descriptor, a second pixel coordinate, and a second camera ID of the monocular feature point in the N target images; Obtaining a hash table based on the matching feature points, the monocular feature points, the first descriptor, the first pixel coordinates, the first camera ID, the second descriptor, the second pixel coordinates, and the second camera ID; The step of obtaining a description subset based on the hash table comprises: Based on the hash table, obtaining a first selected descriptor corresponding to the matching feature point and a second selected descriptor corresponding to the monocular feature point; Determining, from the first selected descriptors, a first result descriptor when the matching feature point is first extracted; Determining, from the second selected descriptor, a second result descriptor when the monocular feature point is extracted for the first time; The descriptor set is obtained according to the matching feature points, the monocular feature points, the first result descriptor and the second result descriptor.

2. The method for obtaining a fused frame according to claim 1, wherein: The step of obtaining a map point set based on the hash table comprises: Based on the hash table, obtaining the selected pixel coordinates corresponding to the matching feature points; Based on the selected pixel coordinates of the matching feature points, obtaining the selected camera coordinates of the matching feature points; Based on the selected camera coordinates of the matching feature points, obtaining depth information of the matching feature points; Based on the depth information of the matching feature points, obtaining the world coordinates of the matching feature points; The map point set is obtained based on the world coordinates of the matching feature points and the matching feature points.

3. The method for obtaining a fused frame according to claim 2, wherein: The step of obtaining the depth information of the matching feature points based on the selected camera coordinates of the matching feature points comprises: determining two result camera coordinates in the selected camera coordinates of the matching feature points; According to the two result camera coordinates, the depth information of the matching feature points is obtained by triangulation and least square method.

4. The method for obtaining a fused frame according to claim 3, wherein: The step of obtaining the world coordinates of the matching feature points based on the depth information of the matching feature points comprises: The world coordinates of the matching feature point are obtained based on the depth information of the matching feature point, the target camera coordinates corresponding to the depth information, the rotation matrix of the target camera coordinate system and the world coordinate system corresponding to the target camera coordinates, and the translation vector between the target camera coordinate system and the world coordinate system.

5. A SLAM system, characterized in that: The SLAM system includes: N cameras, a memory, a processor, and a fusion frame acquisition program stored in the memory and running on the processor. When the fusion frame acquisition program is executed by the processor, the steps of the fusion frame acquisition method as described in any one of claims 1 to 4 are implemented; wherein N is a positive integer greater than or equal to 2.

6. A storage medium, characterized in that: The storage medium stores a fused frame obtaining program, and when the fused frame obtaining program is executed by a processor, the steps of the fused frame obtaining method according to any one of claims 1 to 4 are implemented.

Images