Map construction method, device, SLAM system and storage medium

Abstract

The present invention discloses a map construction method, which is applied to a SLAM system including N cameras, and the method includes the following steps: obtaining a target image of a target area through the N cameras, where N is a positive integer greater than or equal to 2; obtaining a fused frame based on the target image, and the target images of the N cameras at the same time correspond to one fused frame; judging whether the fused frame meets a first preset condition; when the fused frame meets the first preset condition, adding the fused frame to a key frame queue; and constructing a map of the target area using the key frame queue. The present invention also discloses a map construction device, a SLAM system, and a storage medium. When the present invention is used to construct a map of a target area, the amount of data processing is small and the mapping efficiency is high.

Description

Technical Field

[0001] The present invention relates to the field of map construction, and in particular to a map construction method, device, SLAM system and storage medium. Background Art

[0002] The SLAM system is used to achieve real-time positioning or map construction. Due to its miniaturization and economy, it is widely used on lightweight platforms such as AR / MR glasses and quad-rotor drones. The SLAM system includes a monocular camera, a stereo vision camera or a depth color camera. The SLAM system obtains multiple target images of the target area at different times through a monocular camera, tracks and matches multiple target images respectively, and completes local mapping processing.

[0003] However, when the existing method uses the target image of the monocular camera to build a map, the mapping efficiency is low. Summary of the invention

[0004] The main purpose of the present invention is to provide a map construction method, device, SLAM system and storage medium, aiming to solve the technical problem of low mapping efficiency when using the target image of a monocular camera using the existing method in the prior art.

[0005] To achieve the above object, the present invention proposes a map construction method, which is applied to a SLAM system including N cameras, and the method comprises the following steps:

[0006] Acquire a target image of a target area through the N cameras, where N is a positive integer greater than or equal to 2;

[0007] According to the target image, a fused frame is obtained, wherein the target images of the N cameras at the same time correspond to one fused frame;

[0008] Determining whether the fused frame meets a first preset condition;

[0009] When the fused frame meets the first preset condition, adding the fused frame to a key frame queue;

[0010] A map of the target area is constructed using the key frame queue.

[0011] Optionally, the fused frame includes matching feature points that meet a preset matching condition and monocular feature points that do not meet the preset matching condition; before the step of determining whether the fused frame meets the first preset condition, the method further includes:

[0012] Determine the monocular feature points in the fused frame that meet the preset matching conditions with the matching feature points in the fused frame at the previous moment as newly added matching feature points;

[0013] Obtaining a result fused frame according to the newly added matching feature points and the fused frame;

[0014] The step of determining whether the fused frame meets the first preset condition comprises:

[0015] Determining whether the resulting fusion frame meets a first preset condition;

[0016] When the fused frame meets the first preset condition, the step of adding the fused frame to the key frame queue includes:

[0017] When the result fusion frame meets the first preset condition, the result fusion frame is added to a key frame queue.

[0018] Optionally, the first preset condition includes a necessary constraint and a three-choice constraint; wherein,

[0019] The necessary constraint is that the total number of matching feature points of the result fusion frame is not less than a first preset threshold;

[0020] The three-choice constraint includes that the number of result fusion frames between the result fusion frame and a previous result fusion frame that meets the first preset condition exceeds a first preset number of frames, or,

[0021] The number of result fusion frames between the result fusion frame and a previous result fusion frame that meets the first preset condition exceeds a second preset number of frames and the number of result fusion frames that meet the second preset condition in the key frame queue is 0, or,

[0022] The number of result fusion frames in the key frame queue that meet the second preset condition does not exceed a second preset threshold;

[0023] The first preset frame number is greater than the second preset frame number.

[0024] Optionally, before the step of determining whether the result fusion frame is a key frame, the method further includes:

[0025] Determine a reference camera among the N cameras, and determine a coordinate system corresponding to the reference camera as a world coordinate system;

[0026] Obtaining intrinsic parameters of the N cameras;

[0027] Obtaining world coordinates and pixel coordinates of all matching feature points in the result fusion frame;

[0028] Using a uniform motion model assumption method, an estimated position and posture of the result fusion frame and the result fusion frame at an initial moment are obtained;

[0029] According to the world coordinate system, the intrinsic parameters, the world coordinates and the pixel coordinates, the estimated pose is iteratively optimized by formula 1 to obtain the result pose of the result fusion frame;

[0030] The step of judging whether the result fusion frame meets the first preset condition comprises:

[0031] Determining whether the result fusion frame with the result posture meets the first preset condition;

[0032] The formula 1 is:

[0033]

[0034]

[0035] Wherein, i is the i-th camera among the N cameras, j is the j-th feature point among all the matching feature points included in the result fusion frame, and k is the k-th moment corresponding to the result fusion frame. is the reprojection error of the j-th feature point in the camera coordinate system corresponding to the i-th camera at the k-th moment, h is the reprojection function, P j is the world coordinate of the j-th feature point, is the depth coordinate of the j-th feature point at the k-th moment in the camera coordinate system corresponding to the i-th camera, is the horizontal coordinate of the j-th feature point at the k-th moment in the camera coordinate system corresponding to the i-th camera, is the ordinate of the j-th feature point at the k-th moment in the camera coordinate system corresponding to the i-th camera, x 、f y The two intrinsic parameters of the i-th camera are respectively, is the estimated rotation matrix of the reference camera at the kth moment in the estimated pose, is the estimated translation vector of the reference camera at the kth moment in the estimated pose, is a matrix obtained by performing SE3 transformation on the estimated rotation matrix and the estimated translation vector, I3 is a third-order unit matrix, is the translation vector from the world coordinate system to the camera coordinate system corresponding to the i-th camera, is the rotation matrix from the world coordinate system to the camera coordinate system corresponding to the i-th camera.

[0036] Optionally, the step of constructing a map of the target area using the key frame queue includes:

[0037] Obtain a local key frame set according to the result fusion frame in the key frame queue;

[0038] Obtaining a local matching feature point set through the local key frame set;

[0039] According to the local key frame set and the local matching feature point set, the result pose of the result fusion frame and the world coordinates of the matching feature points included in the result fusion frame are updated by formula 2 to obtain an updated result pose and an updated world coordinate;

[0040] obtaining a selected fused frame based on the result fused frame having the updated result pose and the updated world coordinates;

[0041] Using the selected fused frames, a map of the target area is constructed;

[0042] The formula 2 is:

[0043]

[0044]

[0045] Among them, ρ is the kernel function, Q ij is the covariance matrix of the camera coordinate system corresponding to the j-th point in the i-th camera, B L is the local key frame set, B F The matching feature point is included and not included in B L The result fusion frame in is a set consisting of N, which is the total number of cameras. The result fusion frame includes a set of all matching feature points.

[0046] Optionally, after the step of constructing a map of the target area using the selected fused frame, the method further includes:

[0047] Screening candidate key frames from all historical selected fusion frames at a time before the selected fusion frame;

[0048] Performing SIM3 transformation on the selected fusion frame and the candidate key frame to obtain a similarity transformation matrix;

[0049] Using the similarity transformation matrix, adjusting the selected fusion frame and the historical selected fusion frame to obtain an adjusted selected fusion frame and an adjusted historical selected fusion frame;

[0050] Correcting the result pose of the adjusted selected fusion frame, the world coordinates of the matching feature points included in the adjusted selected fusion frame, the result pose of the adjusted historical selected fusion frame, and the world coordinates of the matching feature points included in the adjusted historical selected fusion frame to obtain a corrected fusion frame pose, a corrected fusion frame world coordinates, a corrected historical fusion frame pose, and a corrected historical fusion frame world coordinates;

[0051] The map of the target area is corrected by using the corrected fusion frame pose, the corrected fusion frame world coordinates, the corrected historical fusion frame pose and the corrected historical fusion frame world coordinates to obtain a result map of the target area.

[0052] Optionally, the step of screening candidate key frames from all historical selected fusion frames at a time before the selected fusion frame includes:

[0053] Calculating bag-of-words scores of the selected fused frame and all historical selected fused frames corresponding to the time before the selected fused frame;

[0054] Selecting a preliminary selected fusion frame that meets a third preset condition from all the historically selected fusion frames;

[0055] Selecting key frames whose bag-of-words scores exceed a preset bag-of-words score threshold in the preliminarily selected fusion frames;

[0056] Calculate the maximum number of words shared by the selected key frame and the selected fused frame;

[0057] The candidate key frames whose number of common words is greater than a preset word number threshold are screened out from the selected key frames.

[0058] In addition, to achieve the above-mentioned purpose, the present invention also proposes a map construction device, which is applied to a SLAM system including N cameras, and the device includes:

[0059] An acquisition module, used for acquiring a target image of a target area through the N cameras, where N is a positive integer greater than or equal to 2;

[0060] An acquisition module, used for acquiring a fused frame according to the target image, wherein the target images of the N cameras at the same time correspond to one fused frame;

[0061] A judging module, used for judging whether the fused frame satisfies a first preset condition;

[0062] An adding module, configured to add the fused frame to a key frame queue when the fused frame meets the first preset condition;

[0063] A construction module is used to construct a map of a target area using the key frame queue.

[0064] In addition, to achieve the above-mentioned purpose, the present invention also proposes a SLAM system, which includes: N cameras, a memory, a processor, and a map building program stored in the memory and running on the processor, and when the map building program is executed by the processor, the steps of the map building method described in any one of the above items are implemented; wherein N is a positive integer greater than or equal to 2.

[0065] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a map construction program is stored, and when the map construction program is executed by a processor, the steps of any of the above-mentioned map construction methods are implemented.

[0066] The technical solution of the present invention adopts a map construction method, which is applied to a SLAM system including N cameras. The method includes the following steps: obtaining a target image of a target area through the N cameras, where N is a positive integer greater than or equal to 2; obtaining a fused frame based on the target image, and the target images of the N cameras at the same time correspond to one fused frame; judging whether the fused frame meets a first preset condition; when the fused frame meets the first preset condition, adding the fused frame to a key frame queue; and using the key frame queue to construct a map of the target area. Since, when constructing a map of the target area, the fused frame that meets the first preset condition is added to the key frame queue, and the map of the target area is constructed using the key frame queue, instead of using all the fused frames to construct a map of the target area, when the present invention is used to construct a map of the target area, the amount of data processing is small and the mapping efficiency is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0068] Figure 1 A schematic diagram of the SLAM system structure of the hardware operating environment involved in the embodiment of the present invention;

[0069] Figure 2 This is a flow chart of a first embodiment of a map construction method of the present invention;

[0070] Figure 3 Hash representation for the present invention;

[0071] Figure 4 is a schematic diagram of the triangulation method;

[0072] Figure 5 It is a structural diagram of the fusion frame;

[0073] Figure 6 This is a structural block diagram of a first embodiment of a map construction device of the present invention;

[0074] Figure 7 This is a comparison chart of the accuracy of the method of the present invention and other map construction methods.

[0075] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] Reference Figure 1 , is a schematic diagram of the SLAM system structure of the hardware operating environment involved in the embodiment of the present invention.

[0078] Typically, the SLAM system includes: a camera 307, at least one processor 301, a memory 302, and a fusion frame acquisition program stored in the memory and executable on the processor, wherein the fusion frame acquisition program is configured to implement the steps of the fusion frame acquisition method as described above.

[0079] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. The processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process operations related to the fusion frame acquisition method, so that the fusion frame acquisition method model can be trained and learned autonomously to improve efficiency and accuracy.

[0080] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 801 to implement the fusion frame acquisition method provided in the method embodiment of the present application.

[0081] Camera 307 includes N cameras, and there may be a rotation angle and a translation distance between the N cameras. The N cameras are used to obtain a target image of the target area. Usually, at the same time, the N cameras simultaneously obtain the target image of the target area. Due to the rotation angle and distance between the multiple cameras, the multiple target images obtained are different. Among them, the N cameras can be high-definition cameras or ordinary cameras, and the cameras in the N cameras can be monocular cameras, stereo vision cameras, and depth color cameras.

[0082] In some embodiments, the SLAM system may also optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected via a bus or a signal line. Each peripheral device may be connected to the communication interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 304, a display screen 305 and a power supply 306.

[0083] The communication interface 303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0084] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 304 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 304 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 304 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0085] The display screen 305 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 305 is a touch display screen, the display screen 305 also has the ability to collect touch signals on the surface or above the surface of the display screen 305. The touch signal can be input to the processor 301 as a control signal for processing. At this time, the display screen 305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 305 can be one, the front panel of the electronic device; in other embodiments, the display screen 305 can be at least two, respectively arranged on different surfaces of the electronic device or in a folding design; in some other embodiments, the display screen 305 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device. Even, the display screen 305 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 305 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode) and the like.

[0086] The power supply 306 is used to power various components in the electronic device. The power supply 306 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 306 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology. Those skilled in the art will understand that Figure 1The structure shown in the figure does not constitute a limitation on the fused frame acquisition device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0087] In addition, an embodiment of the present invention further proposes a storage medium, on which a fusion frame acquisition program is stored, and when the fusion frame acquisition program is executed by a processor, the steps of the fusion frame acquisition method as described above are implemented. Therefore, no further description will be given here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in the present application, please refer to the description of the method embodiment of the present application. Determined as an example, the program instructions can be deployed to execute on a SLAM system, or on multiple SLAM systems located at one location, or, on multiple SLAM systems distributed at multiple locations and interconnected by a communication network.

[0088] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0089] Based on the above hardware structure, an embodiment of the map construction method of the present invention is proposed.

[0090] Reference Figure 2 , Figure 2 The flowchart of the first embodiment of the map construction method of the present invention is as follows; the method is applied to a SLAM system, and the method comprises the following steps:

[0091] Step S11: Acquire a target image of a target area through the N cameras, where N is a positive integer greater than or equal to 2.

[0092] It should be noted that the target area may refer to the area that the SLAM system passes through when it is running, and the characteristic information included in the target area can be captured by multiple cameras of the SLAM system. The target area may be an unknown area or a known area, and the present invention does not limit this.

[0093] The target images are target images acquired by N cameras at different times. At the same time, one camera corresponds to one target image, and all target images at the same time are a group of target images. For example, N cameras are 4 cameras, the different times are 3 different times, the target images acquired by the N cameras are 12 target images, and 4 target images at the same time are a group of target images.

[0094] In a specific application, in order to ensure the integrity of the composition of the target area, it is preferred that different moments are continuous moments within a time period be finally selected, and the user can also make other settings according to his or her own needs, and this invention does not limit this.

[0095] Step S12: obtaining a fused frame according to the target image, wherein the target images of the N cameras at the same time correspond to one fused frame.

[0096] A fused frame is obtained using a group of target images at the same time. For example, N cameras are 4 cameras, the different times are 3 different times, the target images acquired by the N cameras are 12 target images, and the 4 target images at the same time are a group of target images. Three fused frames are obtained respectively using the 4 target images at the same time among the 3 different times.

[0097] It can be understood that the method for obtaining the fused frame can be any method based on the feature points of the target image, and this method is not limited to that method. Preferably, the fused frame is obtained using the following method.

[0098] Step 1: Extract features from N target images at the same time to obtain a set of feature points.

[0099] It should be noted that in image processing, feature points refer to points where the grayscale value of an image changes dramatically or points with large curvature on the edge of an image. Features can be divided into color features and texture features. Among them, color features are a global feature that describes the surface properties of the scene corresponding to an image or an image region. Generally, color features are based on pixel features. At this time, all pixels belonging to an image or an image region have their own contributions. Since color is insensitive to changes in the direction, size, etc. of an image or an image region, color features cannot capture the local features of objects in an image well. In addition, when only color features are used for querying, if the database is large, many unnecessary images will often be retrieved. Color histogram is the most commonly used method to express color features. Its advantage is that it is not affected by image rotation and translation changes. Further, with the help of normalization, it is not affected by changes in image scale. The basic disadvantage is that it does not express the characteristic information of color space distribution. Texture features are also a global feature that also describes the surface properties of the scene corresponding to an image or an image region.

[0100] In a specific application, the feature points may be feature points extracted by any of the orb, surf, or sift feature extraction algorithms, or feature points extracted by other methods, and the present invention does not limit this.

[0101] Among them, the feature point includes the camera ID, descriptor and pixel coordinates (2D coordinates) of the feature point; the descriptor can be a vector describing the pixel information around the feature point, and the vector can be a binary number. The camera ID refers to the number of the feature point in the target image. Since the same feature point may be extracted in N target images at the same time, the same feature point can correspond to N camera IDs at the same time; the descriptor refers to the descriptor of the feature point in the target image. The descriptors of the same feature point in different target images at the same time may be different. The pixel coordinates refer to the pixel coordinates of the feature point in the target image. The pixel coordinates of the same feature point in different target images at the same time may be different.

[0102] It can be understood that, for ease of understanding, the target points mentioned below all refer to physical points that objectively exist in the target area and correspond to the feature points.

[0103] For example, the SLAM system has four cameras, camera a, camera b, camera c and camera d. At the same time, the four cameras capture four target images of the target area, perform feature extraction on the four target images, and extract feature points A corresponding to the target area. In the target image of camera a, the feature point camera ID is 12, the feature point A descriptor is 1010100, and the pixel coordinates are (13, 11). In the target image of camera b, the feature point camera ID is 22, the feature point A descriptor is 1010110, and the pixel coordinates are (23, 11). In the target image of camera c, the feature point camera ID is 32, the feature point A descriptor is 1010111, and the pixel coordinates are (33, 11). In the target image of camera d, the feature point camera ID is 32, the feature point A descriptor is 1010101, and the pixel coordinates are (43, 11). At this time, there are 4 descriptors corresponding to feature point A, 4 camera IDs, and 4 pixel coordinates.

[0104] According to the extracted feature points, a feature point set including the feature points and feature information of the feature points is obtained, where the feature information of the feature points is the camera ID, pixel coordinates and descriptors.

[0105] Step 2: Based on the feature point set obtained in the first step, a hash table is obtained.

[0106] Specifically, the second step includes: determining matching feature points that meet preset matching conditions from the feature point set; determining monocular feature points that do not meet the preset matching conditions from the feature point set; and obtaining a hash table based on the matching feature points and the monocular feature points.

[0107] It should be noted that when the descriptors of multiple feature points are similar, the multiple feature points meet the preset matching conditions. The present invention does not limit the degree of similarity of the descriptors, and users can set it according to their own needs. However, in order to ensure that the accuracy of the obtained matching feature points is high, it is better to set the degree of similarity of the descriptors of the preset matching conditions to be higher; for example, the descriptor is a 7-bit binary number, and the preset matching condition is that the first five bits of the descriptor are the same, wherein the preset matching conditions described below are the same as the preset matching conditions here.

[0108] Furthermore, the step of obtaining a hash table based on the matching feature points and the monocular feature points in the second step includes: obtaining a first descriptor, a first pixel coordinate, and a first camera ID of the matching feature points in the N target images; obtaining a second descriptor, a second pixel coordinate, and a second camera ID of the monocular feature points in the N target images; and obtaining a hash table based on the matching feature points, the monocular feature points, the first descriptor, the first pixel coordinate, the first camera ID, the second descriptor, the second pixel coordinate, and the second camera ID.

[0109] It should be noted that after the feature points are extracted, the feature points extracted from the target images corresponding to different cameras at the same time are taken as a set, that is, N cameras correspond to N sets. All feature points in the N sets are matched, and the feature points that meet the preset matching conditions are combined into a matching feature point. The matching feature point corresponds to multiple camera IDs, multiple descriptors, and multiple pixel coordinates as the feature information of the matching feature point, wherein not all target images of each camera can extract matching feature points, but only the target images of at least two cameras can extract matching feature points, so the multiple here refers to a value not greater than N; the feature points in the N sets that do not meet the preset matching conditions are taken as monocular feature points, and the feature information of the monocular feature points is a camera ID, a descriptor, and a pixel coordinate corresponding to the feature point.

[0110] For example, the SLAM system has four cameras, camera a, camera b, camera c and camera d. At the same time, the four cameras capture four target images of the target area, and feature extraction is performed on the four target images. The feature point A corresponding to the target point in the target area is extracted. In the target image of camera a, the camera ID of feature point A is 12, the descriptor is 1010100, and the pixel coordinates are (13, 11). In the target image of camera b, the camera ID of feature point A is 22, the descriptor is 1010110, and the pixel coordinates are (23, 11). In the target image of camera c, the camera ID of feature point A is 32, the descriptor is 101011 1, the pixel coordinates are (33, 11). In the target image of camera d, the camera ID of feature point A is 32, the descriptor is 1010101, and the pixel coordinates are (43, 11). At this time, feature point A is a matching feature point. There are 4 corresponding descriptors, 4 camera IDs, and 4 pixel coordinates. The feature information of matching feature point A includes four camera IDs - 12, 22, 32 and 42, 4 descriptors - 1010100, 1010110, 1010111 and 1010101, and 4 pixel coordinates - (13, 11), (23, 11), (33, 11) and (43, 11).

[0111] Reference Figure 3 , Figure 3 For the hash table scheme of the present invention, all matching feature points and monocular feature points are assigned values ​​of 0 to (M-1), that is, the sum of matching feature points and monocular feature points is M, and the key value id (key) of the hash table is assigned, one matching feature point or one monocular feature point corresponds to one key value, and the feature information of the matching feature point and the monocular feature point is used as the value of the hash table. Wherein, the SLAM system includes N cameras, so the value of a key value in the hash table includes N different feature information, that is, a target point corresponds to a feature information of a feature point in N target images of N cameras, and the value of the hash table is the camera ID, and the descriptor and pixel coordinates are attached to the corresponding camera ID in the mode of sub-feature information; wherein, the value of the hash table can be the camera ID attached to the descriptor and pixel coordinates, or it can be the information obtained by combining the descriptor, pixel coordinates and camera ID, and the present invention does not limit it.

[0112] It can be understood that the first value of a key value (i.e., a feature point corresponding to a target point, which can be a monocular feature point or a matching feature point) is the corresponding camera ID when the feature point is extracted from the target image of the first camera, and the camera corresponding to the value can be determined by the position of the value; in addition, when the feature point cannot be extracted in a certain target image, the camera ID of the feature point in the camera corresponding to the target image is assigned -1, and the camera ID is not attached to the descriptor and pixel coordinates, wherein the camera ID is assigned -1, which means that the feature point is not extracted in the target image of the camera, and the assignment can also be in other forms, such as -2, and the present invention is not limited to -1.

[0113] exist Figure 3 There are N feature information of the feature point corresponding to the key value 104, which are, from left to right, not extracted in the target image of camera 1...extracted in the target image of camera i, and the corresponding camera ID is 201; extracted in the target image of camera N, and the corresponding camera ID is 3.

[0114] It is understandable that the hash table diagram provided by the present invention is a preferred choice, and other types of hash tables can also be determined based on the ideas of the present invention, which will not be described in detail here.

[0115] Step 3: Based on the hash table obtained in the second step, obtain the map point set.

[0116] Specifically, the third step includes: based on the hash table, obtaining the selected pixel coordinates corresponding to the matching feature point; based on the selected pixel coordinates of the matching feature point, obtaining the selected camera coordinates of the matching feature point; based on the selected camera coordinates of the matching feature point, obtaining the depth information of the matching feature point; based on the depth information of the matching feature point, obtaining the world coordinates of the matching feature point; based on the world coordinates of the matching feature point and the matching feature point, obtaining the map point set.

[0117] It should be noted that, referring to the structure of the above hash table, through the hash table key value, that is, the assignment of the feature point, the feature information of the matching feature point corresponding to the key value is found, and the camera to which the feature information belongs is determined as the selected camera, and the camera coordinate system to which the selected camera belongs is the selected camera coordinate system; it can be understood that when a value Q of a key value P in the hash table is -1, the camera to which the value Q corresponding to the key value P belongs is not the selected camera; the pixel coordinates of the matching feature point in the target image of the selected camera are the selected pixel coordinates, and the corresponding normalized selected camera coordinates are obtained using the selected pixel coordinates. Based on the selected camera coordinates of the matching feature point, the depth information of the matching feature point is obtained; based on the depth information of the matching feature point, the world coordinates of the matching feature point are obtained; based on the world coordinates of the matching feature point and the matching feature point, the map point set is obtained.

[0118] Among them, the depth information refers to the depth information of the matching feature point in the camera coordinate system to which the selected camera belongs. It can be understood that only one selected camera coordinate and a corresponding depth information are needed to obtain the world coordinates of the corresponding matching feature point; that is, for all matching feature points, it is sufficient to obtain one depth information for each matching feature point, and there is no need to obtain the depth information of all the selected pixel coordinate systems of the matching feature point.

[0119] For example, there are 4 cameras, and feature point A is extracted from the target images of 3 cameras among the 4 cameras, that is, feature point A is a matching feature point. In the obtained hash table, the key value of matching feature point A is 15, and there are 4 corresponding values, one of which is -1, that is, the matching feature point A is not extracted from the target image of the camera to which the -1 value belongs; then, the cameras to which the other 3 values ​​belong are selected cameras, and the pixel coordinates corresponding to the matching feature point A in the selected camera are the selected pixel coordinates, and the camera coordinates corresponding to the selected pixel coordinates are the selected camera coordinates. Among them, the 3 selected camera coordinates are H, J, and L. When the world coordinates are obtained using the 3 camera coordinates, it is only necessary to use the depth information (l) of any selected camera coordinate (for example, L) among the 3 selected camera coordinates, and the world coordinates of feature point A can be obtained using l and L.

[0120] In addition, the corresponding selected camera coordinates are obtained through formula 3 and the selected pixel coordinates.

[0121] The formula three is:

[0122]

[0123]

[0124] Among them, u is the horizontal coordinate in the selected pixel coordinates, v is the vertical coordinate in the selected pixel coordinates, is the horizontal coordinate of the selected camera coordinates, is the ordinate of the selected camera coordinates, f x , f y , c x , c y are all the internal parameters of the selected camera; it is understandable that the internal parameters of the same feature point corresponding to different cameras may be different; the coordinates obtained according to formula 1 only include the horizontal coordinate and the vertical coordinate Therefore, the camera coordinates at this time are normalized camera coordinates, expressed as

[0125] Furthermore, the step of obtaining the depth information of the matching feature points based on the selected camera coordinates of the matching feature points includes: determining two result camera coordinates in the selected camera coordinates of the matching feature points; and obtaining the depth information of the matching feature points by triangulation and least squares method based on the two result camera coordinates.

[0126] It can be understood that there are at least two feature information corresponding to the matching feature points, namely Figure 3 In the hash table shown, there are at least two non-1 values ​​corresponding to the matching feature points; when the number of matching feature points is large, that is, when the key values ​​corresponding to the matching feature points in the hash table are large in number and non-1, feature information corresponding to any two non-1 values ​​is determined as result feature information, and the camera coordinates obtained using the pixel coordinates in the result feature information are used as result camera coordinates, and the camera to which the result camera coordinates of the matching feature points belong is the result camera.

[0127] It should be noted that the formula of the triangulation method is as follows, that is, Formula 4:

[0128] s2X2=s1RX1+t

[0129] Among them, X1 is one of the two resulting camera coordinates, X2 is the other of the two resulting camera coordinates, s1 is the depth information of X1 in the corresponding camera coordinate system, s2 is the depth information of X2 in the corresponding camera coordinate system, R is the rotation matrix of the camera coordinate system corresponding to X1 and the camera coordinate system corresponding to X2, and t is the translation vector of the camera coordinate system corresponding to X1 and the camera coordinate system corresponding to X2. It can be understood that the camera coordinate X can be expressed as above

[0130] Reference Figure 4 , Figure 4It is a schematic diagram of the triangulation method, P is the original position map corresponding to a feature point in the target image of the SLAM system, that is, the actual position of the feature point in the world coordinates, I1 is the target image in the result camera, P1 is the coordinate pixel of P in I1, I2 is the target image in another result camera, P2 is the coordinate pixel of P in I2.

[0131] In specific applications, formula 4 and the least square method are used to obtain the depth information of any camera coordinate in the two result camera coordinates. The result camera coordinate corresponding to the depth information is the target camera coordinate. For example, when s1 in formula 2 is used as the depth information, X1 corresponding to the depth information is the target camera coordinate. The depth information is used to restore the original camera coordinate corresponding to the target camera coordinate, that is, the target camera coordinate The horizontal coordinate, vertical coordinate and depth in are multiplied by the depth information Z, respectively, where the original camera coordinates are the coordinates in the camera coordinate system to which the target camera coordinates belong. For example, X1 is the result camera coordinate of feature point A of camera 1, and the depth information s1 of X1 is used as the selected depth information to obtain the original camera coordinates (X, Y, Z) of X1, which are the coordinates in the camera coordinate system to which camera 1 belongs.

[0132] Furthermore, the step of obtaining the world coordinates of the matching feature points based on the depth information of the matching feature points includes: obtaining the world coordinates of the matching feature points based on the depth information of the matching feature points, the target camera coordinates corresponding to the depth information, the rotation matrix of the target camera coordinate system and the world coordinate system corresponding to the target camera coordinates, and the translation vector between the target camera coordinate system and the world coordinate system.

[0133] It can be understood that for each matching feature point, only one depth information needs to be obtained, and the original camera coordinates corresponding to the feature point are obtained based on the depth information and the target camera coordinates corresponding to the depth information. Among them, the camera coordinate system to which the original camera belongs obtained based on the target camera coordinates is the target camera coordinate system. For example, X1 is one of the result camera coordinates of feature point A of camera No. 1, and the depth information s1 of X1 is selected as the depth information (at this time, the result camera coordinate X1 is the target camera coordinate), and the original camera coordinates (X, Y, Z) of X1 are obtained. The original camera coordinates are the coordinates in the camera coordinate system to which camera No. 1 belongs, and the camera coordinate system to which camera No. 1 belongs is the target camera coordinate system. Then, according to the rotation matrix of the target camera coordinate system and the world coordinate system and the translation vector of the target camera coordinate system and the world coordinate system, the world coordinates of the matching feature point are obtained, and the world coordinates of the original camera coordinates are obtained. The world coordinates of the original camera coordinates are the world coordinates of the feature point.

[0134] In a specific application, the leftmost camera among the N cameras in the SLAM system can be used as camera No. 1, and the coordinate system of camera No. 1 at the initial moment can be used as the world coordinate system. The world coordinate system can also be determined in other ways, which is not limited by the present invention.

[0135] When using the above-mentioned hash table to obtain the world coordinates of the matching feature points, the key value of the hash table is used as the map point element of the map point set, the world coordinates of the matching feature points corresponding to the key value are used as the world coordinates of the key value, and the world coordinates of the monocular feature points corresponding to the key value are assigned to 0. According to the key value and the world coordinates, the map point set is obtained. It can be understood that if the number of key values ​​is M, the number of map points included in the map point set is also M, wherein the world coordinates of the map points corresponding to the monocular feature points are 0, and the world coordinates of the map points corresponding to the matching feature points are not 0 and are meaningful, wherein the map points are defined as matching feature points or monocular feature points with world coordinates.

[0136] Step S15: Based on the hash table obtained in the second step, obtain a description subset.

[0137] Further, step S15 includes: based on the hash table, obtaining a first selected descriptor corresponding to the matching feature point and a second selected descriptor corresponding to the monocular feature point; determining a first result descriptor when the matching feature point is first extracted in the first selected descriptor; determining a second result descriptor when the monocular feature point is first extracted in the second selected descriptor; and obtaining the descriptor set according to the matching feature point, the monocular feature point, the first result descriptor and the second result descriptor.

[0138] It should be noted that the key value of each feature point in the hash table corresponds to multiple values, and each value includes feature information (descriptor, camera ID and pixel coordinates) of the feature point under different cameras. For the matching feature points, the key value corresponds to multiple feature information, and the key value of the monocular feature point corresponds to one feature information; in the matching feature points, the valid feature information (that is, the feature information corresponding to the camera ID whose value is not -1) is determined as the selected feature information, and in the selected feature information, the descriptor in the target image corresponding to the first time the matching feature point is extracted is determined as the first result descriptor; in the monocular feature points, the only descriptor is determined as the second result descriptor. It can be understood that the descriptor must be the descriptor corresponding to the first extraction of the feature point.

[0139] The key value in the hash table is used as the element unit of the description subset, and the corresponding first result descriptor or second result descriptor is used as the selected descriptor of the key value. The description subset is obtained according to the key value, the first result descriptor and the second result descriptor. It can be understood that if the number of key values ​​is M, the number of selected descriptors is also M, wherein the sum of the matching feature points and monocular feature points corresponding to the M key values ​​is M.

[0140] Step S16: Obtain a fused frame based on the hash table obtained in the second step, the map point set obtained in the third step, and the description subset obtained in the fourth step.

[0141] Reference Figure 4 , Figure 4 is a structural diagram of a fusion frame, wherein the fusion frame is obtained based on the target images of N cameras at a certain moment, and the shapes of the frames (i.e., target images) corresponding to the cameras (representing the angles and positions of the target images) are different, wherein the pose refers to the different angles and displacements, and the virtual fusion frame is the fusion frame, including a map point set, a hash table, and a description subset, wherein the sum of matching feature points and monocular feature points is M, the number of map points in the corresponding map point set is M, and the number of descriptors in the description subset is also M; the map point set originates from the map points, schematically, the map points are 5 map points - P1, P2, P3, P4 and P5, and the map points originate from the feature points, schematically, the feature points include 9 points, and there are matching feature points and monocular feature points among the 9 points.

[0142] When fused frames at different times are obtained, the monocular feature points in the fused frame that meet the preset matching conditions with the matching feature points in the fused frame at the previous time are determined as newly added matching feature points; the feature points of the fused frame are updated using the newly added matching feature points to obtain a result fused frame, and all matching feature points of the result fused frame are the sum of the matching feature points of the fused frame and the newly added matching feature points. The preset matching conditions refer to the description of the above preset matching conditions.

[0143] For example, at the kth moment, the fused frame includes 350 matching feature points and 250 monocular feature points. At the (k-1)th moment, the matching feature points are 330 and 270 monocular feature points. The 250 matching feature points at the kth moment are matched with the 330 matching feature points at the (k-1)th moment, and 30 points are matched. Therefore, 30 new matching feature points are added, and the resulting fused frame at the kth moment includes 380 matching feature points.

[0144] Step S13: Determine whether the fused frame meets a first preset condition.

[0145] It should be noted that the first preset condition includes a necessary constraint and a three-choice constraint; wherein the necessary constraint is that the total number of matching feature points of the result fusion frame is not less than a first preset threshold; the three-choice constraint includes that the number of result fusion frames between the result fusion frame and the previous result fusion frame that meets the first preset condition exceeds the first preset number of frames, or, the number of result fusion frames between the result fusion frame and the previous result fusion frame that meets the first preset condition exceeds the second preset number of frames and the number of result fusion frames that meet the second preset condition in the key frame queue is 0, or, the number of result fusion frames that meet the second preset condition in the key frame queue does not exceed the second preset threshold; the first preset number of frames is greater than the second preset number of frames.

[0146] It can be understood that the first preset condition includes the necessary condition that the total number of matching feature points of the result fusion frame is not less than the first preset threshold, and the three-selection condition of the first preset condition is that the number of result fusion frames between the result fusion frame and the previous result fusion frame that meets the first preset condition exceeds the first preset number of frames, or, the number of result fusion frames between the result fusion frame and the previous result fusion frame that meets the first preset condition exceeds the second preset number of frames and the number of result fusion frames that meet the second preset condition in the key frame queue is 0, or, the number of result fusion frames that meet the first preset condition in the key frame queue does not exceed one of the second preset thresholds. Wherein, the first preset threshold and the second preset threshold are both thresholds set by the user according to their own needs, and the present invention does not limit them. The second preset threshold number is preferably selected as 3; the first preset number of frames is the maximum number of fusion frames between adjacent result fusion frames that meet the first preset condition, and the second preset number of frames is the minimum number of fusion frames between adjacent result fusion frames that meet the first preset condition. The maximum and minimum values ​​can be set by the user according to their own needs; the second preset condition can refer to the fused key frames in the key frame queue that are not used for composition and are in the queue.

[0147] Specifically, when the number of result fusion frames between the result fusion frame and the previous result fusion frame that meets the first preset condition exceeds the first preset number of frames, the result fusion frame is determined to meet the first preset condition to avoid the interval between the result fusion frames being too far and the amount of information being lost, and at the same time avoid the interval between the result fusion frames being too close and there being a lot of information overlap; at the same time, when the number of result fusion frames between the result fusion frame and the previous result fusion frame that meets the first preset condition exceeds the second preset number of frames and the number of result fusion frames that meet the second preset condition in the key frame queue is 0, the current key frame queue is idle, and the result fusion frame can be used as the result fusion frame that meets the first preset condition to avoid the interval between the result fusion frames being too short and there being excessive overlap of information.

[0148] Furthermore, the method before step S13 also includes: determining a reference camera among the N cameras, and determining the coordinate system corresponding to the reference camera as the world coordinate system; obtaining the intrinsic parameters of the N cameras; obtaining the world coordinates and pixel coordinates of all matching feature points in the result fusion frame; using the uniform motion model hypothesis method, obtaining the estimated pose of the result fusion frame and the result fusion frame at the initial moment; according to the world coordinate system, the intrinsic parameters, the world coordinates and the pixel coordinates, the estimated pose is iteratively optimized through formula 1 to obtain the result pose of the result fusion frame. The formula 1 is:

[0149]

[0150]

[0151] Wherein, i is the i-th camera among the N cameras, j is the j-th feature point among the matching feature points included in the result fusion frame, and k is the k-th moment corresponding to the result fusion frame. The reprojection error of the j-th feature point in the camera coordinate system corresponding to the i-th camera at the k-th time, h is the reprojection function, P j is the world coordinate of the j-th feature point, is the depth coordinate of the j-th feature point at the k-th moment in the camera coordinate system corresponding to the i-th camera, is the horizontal coordinate of the j-th feature point at the k-th moment in the camera coordinate system corresponding to the i-th camera, is the ordinate of the j-th feature point at the k-th moment in the camera coordinate system corresponding to the i-th camera, x 、f y The two intrinsic parameters of the i-th camera are respectively, is the estimated rotation matrix of camera No. 1 at the kth moment in the estimated pose, is the estimated translation vector of camera No. 1 at the kth moment in the estimated pose, is a matrix obtained by performing SE3 transformation on the estimated rotation matrix and the estimated translation vector, I3 is a third-order unit matrix, is the translation vector from the world coordinate system to the camera coordinate system corresponding to the i-th camera, is the rotation matrix from the world coordinate system to the camera coordinate system corresponding to the i-th camera.

[0152] Correspondingly, step S13 includes: determining whether the result fusion frame with the result posture meets the first preset condition.

[0153] It should be noted that the reference camera of the present invention can be the leftmost camera among the N cameras, or a camera at another position, and the present invention does not make any limitation thereto. The camera coordinate system to which the reference camera belongs at the initial moment is the world coordinate system.

[0154] At the same time, based on the uniform motion model method, the pose change of the result fusion frame corresponding to the first two moments of the result fusion frame is multiplied by the pose of the result fusion frame at the previous moment of the result fusion frame to obtain the estimated pose of the result fusion frame.

[0155] Step S14: When the fused frame meets the first preset condition, the fused frame is added to a key frame queue.

[0156] Further, step S14 includes: when the result fusion frame meets the first preset condition, adding the result fusion frame to a key frame queue.

[0157] It should be noted that the key frame queue is used to store the result fusion frames that meet the first preset condition. Since more data processing is required when constructing the map of the target area, when the result fusion frame that meets the first preset condition is obtained, the result fusion frame that meets the first preset condition cannot be used to construct the map of the target area immediately. The result fusion frame that meets the first preset condition needs to be used to construct the map of the target area in sequence, and the newly obtained result fusion frame that meets the first preset condition needs to be added to the key frame queue, so as to use the result fusion frame in the key frame queue to construct the map of the target area.

[0158] Usually, there are always a certain number of result fusion frames that meet the first preset condition in the key frame queue. When the result fusion frames meet the first preset condition, they are immediately used to construct a map of the target area.

[0159] Step S15: construct a map of the target area using the key frame queue.

[0160] It should be noted that, according to the result fusion frame in the key frame queue, a local key frame set is obtained; through the local key frame set, a local matching feature point set is obtained; according to the local key frame set and the local matching feature point set, the result pose of the result fusion frame and the world coordinates of the matching feature points included in the result fusion frame are updated by formula 2 to obtain an updated result pose and an updated world coordinate; based on the result fusion frame having the updated result pose and the updated world coordinate, a selected fusion frame is obtained; and a map of the target area is constructed using the selected fusion frame;

[0161] The formula 2 is:

[0162]

[0163]

[0164] Among them, ρ is the kernel function, Q ij is the covariance matrix of the camera coordinate system corresponding to the j-th point in the i-th camera, B L is the local key frame set, B F The matching feature point is included and not included in B L The result fusion frame in is a set consisting of N, where N is the total number of cameras. The result fusion frame includes a set of all matching feature points.

[0165] The result fusion frame is taken out from the key frame queue in order (the result fusion frame satisfies the first preset condition), and the association between the currently taken out result fusion frame and all the previously taken out result fusion frames is established, and the association between the currently taken out result fusion frame and the matching feature points in the currently taken out result fusion frame is established; among all the matching feature points included in all the taken out result fusion frames, the matching feature points that appear less than 3 times in total are eliminated, and then, the repeated matching feature points in the currently taken out result fusion frame and the adjacent taken out result fusion frames are eliminated, and the currently taken out result fusion frame and the adjacent taken out result fusion frames are taken together as a local key frame set, and all the matching feature points in the set are the local matching feature point set.

[0166] The data in the result fusion frame is updated using the obtained updated result pose and the updated world coordinates, that is, the previous result pose and world coordinates are replaced by the updated result pose and world coordinates to obtain the selected fusion frame, and the matching feature points in the selected fusion frame that exist in the common view selected fusion frame are eliminated to obtain the composition points, wherein the elimination ratio is preferably 90%, wherein the common view selected fusion frame is a selected key frame in which the number of consensus map points in all the selected fusion frames obtained at the current moment exceeds 15, and the consensus map point refers to the matching feature point with the same world coordinate.

[0167] Finally, the map of the target area is constructed using the composition points. It can be understood that the composition points obtained after screening each selected fusion frame are the matching feature points corresponding to the new target points captured by the SLAM system (i.e. the target points mentioned above: the physical points corresponding to the feature points that objectively exist in the target area), and the map of the target area is constructed using the matching feature points.

[0168] In a specific application, as time goes by, N target images corresponding to each moment are used to obtain fused frames, and a map of the target area is constructed using the composition points corresponding to all fused frames during the entire running time.

[0169] The technical solution of this embodiment adopts a map construction method, which is applied to a SLAM system including N cameras, and obtains a target image of the target area through the N cameras; obtains a fused frame based on the target image, and the target images of the N cameras at the same time correspond to one fused frame; determines whether the fused frame meets a first preset condition; when the fused frame meets the first preset condition, adds the fused frame to a key frame queue; and uses the key frame queue to build a map of the target area. Because when building a map of the target area, the fused frame that meets the preset condition is added to the key frame queue, and the map of the target area is built using the key frame queue, instead of using all the fused key frames to build a map of the target area, when the present invention is used to build a map of the target area, the amount of data processing is small and the map building efficiency is high.

[0170] At the same time, Formula 1 and Formula 2 are used to optimize the information included in the fusion frame, and the optimized result fusion frame is used to obtain the result map of the target area. The result map has a high accuracy rate.

[0171] Furthermore, after step S15, it also includes: screening candidate key frames from all historical selected fusion frames before the selected fusion frame; performing SIM3 transformation on the selected fusion frame and the candidate key frames to obtain a similarity transformation matrix; using the similarity transformation matrix, adjusting the selected fusion frame and the historical selected fusion frame to obtain an adjusted selected fusion frame and an adjusted historical selected fusion frame; correcting the result pose of the adjusted selected fusion frame, the world coordinates of the matching feature points included in the adjusted selected fusion frame, the result pose of the adjusted historical selected fusion frame, and the world coordinates of the matching feature points included in the adjusted historical selected fusion frame to obtain a corrected fusion frame pose, a corrected fusion frame world coordinates, a corrected historical fusion frame pose, and a corrected historical fusion frame world coordinates; using the corrected fusion frame pose, the corrected fusion frame world coordinates, the corrected historical fusion frame pose, and the corrected historical fusion frame world coordinates, to correct the map of the target area to obtain a result map of the target area.

[0172] It should be noted that, each time after constructing a map of the target area using the selected fused frame having the updated result pose and the updated world coordinates, the selected fused frame needs to be subjected to the above correction processing to obtain a result map.

[0173] In a specific application, during correction, the above formula 2 is used to correct the result pose of the adjusted result fusion frame, the world coordinates of the matching feature points included in the adjusted result fusion frame, the result pose included in the adjusted historical result fusion frame, and the world coordinates of the matching feature points included in the adjusted historical result fusion frame to obtain the corrected fusion frame pose, the corrected fusion frame world coordinates, the corrected historical fusion frame pose, and the corrected historical fusion frame world coordinates; wherein the pose includes a rotation matrix and a translation vector. No further details are given here.

[0174] Furthermore, the step of screening candidate key frames from all historical selected fusion frames before the selected fusion frame includes: calculating the bag-of-words score of the selected fusion frame and all historical selected fusion frames corresponding to the time before the selected fusion frame; screening preliminary fusion frames that meet the third preset condition from all historical selected fusion frames; screening selected key frames whose bag-of-words score exceeds a preset bag-of-words score threshold from the preliminary fusion frames; calculating the maximum value of the number of words shared by the selected key frame and the selected fusion frame; and screening candidate key frames whose number of shared words is greater than a preset word number threshold from the selected key frames.

[0175] The third precondition is implemented as follows:

[0176] The selected fused frames connected to the selected fused frames having the updated result pose and the updated world coordinates constitute a "sub-candidate group", and then the selected fused frames themselves are also added to the "sub-candidate group"; detect whether each selected fused frame in the "sub-candidate group" exists in the "continuous group", if so, record the value of the current continuous variable plus 1, and put the "sub-candidate group" into the current "continuous group"; wherein, the step of detecting whether each selected fused frame in the "sub-candidate group" exists in the "continuous group" is as follows: traverse each "sub-candidate group", detect whether each selected fused frame in each "sub-candidate group" exists in the "continuous group", if there is a frame that exists in both the sub-candidate group and the previous "continuous group", then the "sub-candidate group" is continuous with the previous "continuous group", that is, there is a "continuous group"; if the value of the current continuous variable is greater than or equal to 3, then the selected fused frame corresponding to the "sub-candidate group" meets the third preset condition.

[0177] In addition, the preset bag-of-words score threshold can be the lowest score of the bag-of-words score, or other thresholds, and the user can set it according to their own needs; the preset word count threshold = the maximum number of common words × the preset coefficient. The present invention preferably uses a preset coefficient of 0.8, and the user can set it according to their own needs, and there is no limitation here.

[0178] Reference Figure 6 , Figure 6This is a structural block diagram of the first embodiment of the map construction device of the present invention; applied to a SLAM system including N cameras, the device includes:

[0179] An acquisition module 10 is used to acquire a target image of a target area;

[0180] An acquisition module 20 is used to obtain a fused frame according to the target image, and the target images of the N cameras at the same time correspond to one fused frame;

[0181] A judging module 30, used to judge whether the fused frame is a key frame;

[0182] An adding module 40, configured to add the fused frame to a key frame queue when the fused frame is a key frame;

[0183] The construction module 50 is used to construct a map of the target area using the key frame queue.

[0184] Reference Figure 7 , Figure 7 This is a comparison chart of the accuracy of the method of the present invention and other map construction methods; wherein the data in the figure includes the accuracy of the result map of the target area obtained by using the method of the present invention using a multi-camera SLAM system, and also includes the accuracy of the result map of the target area constructed using the open source framework ORB-SLAM2 and VINS-Stereo.

[0185] The above descriptions are only optional embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings, or directly / indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A map construction method, characterized in that: Applied to a SLAM system including N cameras, the method comprises the following steps: Acquire a target image of a target area through the N cameras, where N is a positive integer greater than or equal to 2; According to the target image, a fused frame is obtained, wherein the target images of the N cameras at the same time correspond to one fused frame; Determining whether the fused frame meets a first preset condition; When the fused frame meets the first preset condition, adding the fused frame to a key frame queue; Using the key frame queue, construct a map of the target area; The fused frame includes matching feature points that meet the preset matching condition and monocular feature points that do not meet the preset matching condition; before the step of determining whether the fused frame meets the first preset condition, the method further includes: Determine the monocular feature points in the fused frame that meet the preset matching conditions with the matching feature points in the fused frame at the previous moment as newly added matching feature points; Obtaining a result fused frame according to the newly added matching feature points and the fused frame; The step of determining whether the fused frame meets the first preset condition comprises: Determining whether the resulting fusion frame meets a first preset condition; When the fused frame meets the first preset condition, the step of adding the fused frame to the key frame queue includes: When the result fusion frame meets the first preset condition, adding the result fusion frame to a key frame queue; The first preset condition includes a necessary constraint and a three-choice constraint; wherein, The necessary constraint is that the total number of matching feature points of the result fusion frame is not less than a first preset threshold; The three-choice constraint includes that the number of result fusion frames between the result fusion frame and a previous result fusion frame that meets the first preset condition exceeds a first preset number of frames, or, The number of result fusion frames between the result fusion frame and a previous result fusion frame that meets the first preset condition exceeds a second preset number of frames and the number of result fusion frames that meet the second preset condition in the key frame queue is 0, or, The number of result fusion frames in the key frame queue that meet the second preset condition does not exceed a second preset threshold; The first preset frame number is greater than the second preset frame number.

2. The map construction method according to claim 1, characterized in that: Before the step of determining whether the result fusion frame satisfies the first preset condition, the method further includes: Determine a reference camera among the N cameras, and determine a coordinate system corresponding to the reference camera as a world coordinate system; Obtaining intrinsic parameters of the N cameras; Obtaining world coordinates and pixel coordinates of all matching feature points in the result fusion frame; Using a uniform motion model assumption method, an estimated position and posture of the result fusion frame and the result fusion frame at an initial moment are obtained; According to the world coordinate system, the intrinsic parameters, the world coordinates and the pixel coordinates, the estimated pose is iteratively optimized by formula 1 to obtain the result pose of the result fusion frame; The step of judging whether the result fusion frame meets the first preset condition comprises: Determining whether the result fusion frame having the result posture satisfies a first preset condition; The formula 1 is: Wherein, i is the i-th camera among the N cameras, j is the j-th feature point among all the matching feature points included in the result fusion frame, and k is the k-th moment corresponding to the result fusion frame. is the reprojection error of the j-th feature point in the camera coordinate system corresponding to the i-th camera at the k-th moment, h is the reprojection function, is the world coordinate of the j-th feature point, is the depth coordinate of the j-th feature point at the k-th moment in the camera coordinate system corresponding to the i-th camera, is the horizontal coordinate of the j-th feature point at the k-th moment in the camera coordinate system corresponding to the i-th camera, is the ordinate of the j-th feature point at the k-th moment in the camera coordinate system corresponding to the i-th camera, , are the two intrinsic parameters of the i-th camera, is the estimated rotation matrix of the reference camera at the kth moment in the estimated pose, is the estimated translation vector of the reference camera at the kth moment in the estimated pose, is a matrix obtained by performing SE3 transformation on the estimated rotation matrix and the estimated translation vector, is the third-order identity matrix, is the translation vector from the world coordinate system to the camera coordinate system corresponding to the i-th camera, is the rotation matrix from the world coordinate system to the camera coordinate system corresponding to the i-th camera.

3. The map construction method according to claim 2, characterized in that: The step of constructing a map of the target area using the key frame queue comprises: Obtain a local key frame set according to the result fusion frame in the key frame queue; Obtaining a local matching feature point set through the local key frame set; According to the local key frame set and the local matching feature point set, the result pose of the result fusion frame and the world coordinates of the matching feature points included in the result fusion frame are updated by formula 2 to obtain an updated result pose and an updated world coordinate; obtaining a selected fused frame based on the result fused frame having the updated result pose and the updated world coordinates; Using the selected fused frames, a map of the target area is constructed; The formula 2 is: in, is the kernel function, is the covariance matrix of the camera coordinate system corresponding to the j-th point in the i-th camera, is the local key frame set, The matching feature points are included and not included in the The result fusion frame in is a set consisting of N, where N is the total number of cameras. The result fusion frame includes a set of all matching feature points.

4. The map construction method according to claim 3, characterized in that: After the step of constructing a map of the target area using the selected fused frame, the method further comprises: Screening candidate key frames from all historical selected fusion frames at a time before the selected fusion frame; Performing SIM3 transformation on the selected fusion frame and the candidate key frame to obtain a similarity transformation matrix; Using the similarity transformation matrix, adjusting the selected fusion frame and the historical selected fusion frame to obtain an adjusted selected fusion frame and an adjusted historical selected fusion frame; Correcting the result pose of the adjusted selected fusion frame, the world coordinates of the matching feature points included in the adjusted selected fusion frame, the result pose of the adjusted historical selected fusion frame, and the world coordinates of the matching feature points included in the adjusted historical selected fusion frame to obtain a corrected fusion frame pose, a corrected fusion frame world coordinates, a corrected historical fusion frame pose, and a corrected historical fusion frame world coordinates; The map of the target area is corrected by using the corrected fusion frame pose, the corrected fusion frame world coordinates, the corrected historical fusion frame pose and the corrected historical fusion frame world coordinates to obtain a result map of the target area.

5. The map construction method according to claim 4, characterized in that: The step of screening candidate key frames from all historical selected fusion frames at the time before the selected fusion frame comprises: Calculating bag-of-words scores of the selected fused frame and all historical selected fused frames corresponding to the time before the selected fused frame; Selecting a preliminary selected fusion frame that meets a third preset condition from all the historically selected fusion frames; Selecting key frames whose bag-of-words scores exceed a preset bag-of-words score threshold in the preliminarily selected fusion frames; Calculate the maximum number of words shared by the selected key frame and the selected fused frame; The candidate key frames whose number of common words is greater than a preset word number threshold are screened out from the selected key frames.

6. A map construction device, characterized in that: Applied to a SLAM system including N cameras, the device includes: An acquisition module, used for acquiring a target image of a target area through the N cameras, where N is a positive integer greater than or equal to 2; An acquisition module, used for acquiring a fused frame according to the target image, wherein the target images of the N cameras at the same time correspond to one fused frame; A judging module, used for judging whether the fused frame satisfies a first preset condition; An adding module, configured to add the fused frame to a key frame queue when the fused frame meets the first preset condition; A construction module, used to construct a map of a target area using the key frame queue; The fused frame includes matching feature points that meet the preset matching conditions and monocular feature points that do not meet the preset matching conditions; The acquisition module is further used to determine the monocular feature points in the fused frame that meet the preset matching conditions with the matching feature points in the fused frame at the previous moment as newly added matching feature points; and obtain a result fused frame according to the newly added matching feature points and the fused frame; When the judging module executes the step of judging whether the fused frame satisfies the first preset condition, it is specifically used to judge whether the resultant fused frame satisfies the first preset condition; When the adding module executes the step of adding the fused frame to the key frame queue when the fused frame meets the first preset condition, the adding module is specifically used to add the result fused frame to the key frame queue when the result fused frame meets the first preset condition; Wherein, the first preset condition includes a necessary constraint and a three-choice constraint; The necessary constraint is that the total number of matching feature points of the result fusion frame is not less than a first preset threshold; The three-choice constraint includes that the number of result fusion frames between the result fusion frame and a previous result fusion frame that meets the first preset condition exceeds a first preset number of frames, or, The number of result fusion frames between the result fusion frame and a previous result fusion frame that meets the first preset condition exceeds a second preset number of frames and the number of result fusion frames that meet the second preset condition in the key frame queue is 0, or, The number of result fusion frames in the key frame queue that meet the second preset condition does not exceed a second preset threshold; The first preset frame number is greater than the second preset frame number.

7. A SLAM system, characterized in that: The SLAM system includes: N cameras, a memory, a processor, and a map building program stored in the memory and running on the processor, wherein the map building program, when executed by the processor, implements the steps of the map building method as described in any one of claims 1 to 5; wherein N is a positive integer greater than or equal to 2.

8. A storage medium, characterized in that: The storage medium stores a map construction program, and when the map construction program is executed by the processor, the steps of the map construction method according to any one of claims 1 to 5 are implemented.

Images