Monocular SLAM initialization method, device, equipment and storage medium

Abstract

The present invention discloses a monocular SLAM initialization method, device, equipment and storage medium, which relates to the field of mobile robot technology. The method includes: obtaining a first frame image and a first feature point of the first frame image; inputting the first frame image into a pre-trained generative network model to obtain a first homography matrix; wherein the generative network model is constructed based on a convolutional neural network and a generative adversarial network; according to the first homography matrix, the first frame image is perspective transformed to obtain a virtual frame image; using the first homography matrix, the first feature point is converted to the virtual frame image to obtain a second feature point of the virtual frame image; according to the second feature point and the corresponding first feature point, a triangulation algorithm is used to obtain a first map point. The present invention solves the problem of low success rate of monocular SLAM initialization in the prior art, and achieves the effect of monocular SLAM initialization with only one video frame image.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robots, and in particular to a monocular SLAM initialization method, device, equipment and storage medium. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) technology is a key technology for realizing autonomous positioning and navigation of robots. Monocular SLAM, whose full name is monocular vision SLAM system, is a system that uses a camera as a sensor to realize autonomous positioning and navigation of robots. It has the advantages of simple system structure, low cost and easy implementation. It is used in AR (Augmented Reality), AI (Artificial Intelligence) and other fields.

[0003] In the current monocular SLAM system, the initialization operation of the monocular SLAM system is generally implemented by selecting two key video frame images with large parallax. The problem with this monocular SLAM initialization method is that two video frames with large parallax are required for initialization. It is difficult for new users to master the correct camera method to guide initialization, and initialization failure is likely to occur. There is a problem of low success rate of monocular SLAM initialization. Summary of the invention

[0004] The main purpose of the present invention is to provide a monocular SLAM initialization method, device, equipment and storage medium, aiming to solve the technical problem of low success rate of monocular SLAM initialization in the prior art.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a monocular SLAM initialization method, the method comprising:

[0007] Acquire a first frame image and a first feature point of the first frame image;

[0008] Inputting the first frame image into a pre-trained generative network model to obtain a first homography matrix; wherein the generative network model is constructed based on a convolutional neural network and a generative adversarial network;

[0009] According to the first homography matrix, performing perspective transformation on the first frame image to obtain a virtual frame image;

[0010] Using the first homography matrix, converting the first feature point to the virtual frame image to obtain a second feature point of the virtual frame image;

[0011] A first map point is obtained by using a triangulation algorithm according to the second feature point and the corresponding first feature point.

[0012] Optionally, in the above monocular SLAM initialization method, after the step of obtaining the first map point by using a triangulation algorithm according to the second feature point and the corresponding first feature point, the method further includes:

[0013] Acquire a second frame image and a third feature point of the second frame image;

[0014] Taking the first frame image as a reference frame, performing feature point matching on the first frame image and the second frame image to obtain first feature points that are successfully matched;

[0015] Using the first homography matrix, projecting the first map point corresponding to the successfully matched first feature point onto the second frame image to obtain a projection point in the second frame image;

[0016] The first map point corresponding to the projection point that meets a preset condition is determined as a final map point, wherein the preset condition is that the third feature point exists within a preset range centered on the projection point.

[0017] Optionally, in the above monocular SLAM initialization method, after the step of determining the first map point corresponding to the projection point that meets the preset conditions as the final map point, the method further includes:

[0018] According to the correspondence between the final map point and the third feature point, the camera pose of the second frame image is obtained by minimizing the reprojection error method.

[0019] Optionally, in the above monocular SLAM initialization method, before the step of inputting the first frame image into a pre-trained generative network model to obtain a first homography matrix, the method further includes:

[0020] Get the training dataset;

[0021] Construct hybrid neural networks;

[0022] According to the training data set, the hybrid neural network is trained to obtain the generated network model;

[0023] Wherein, the hybrid neural network comprises:

[0024] The feature extraction layer is used to extract features from the input image and output feature vectors;

[0025] A convolutional network layer, used for obtaining a generated image of the input image by upsampling according to the feature vector, the convolutional network layer comprising a convolution module, an activation module and a custom module;

[0026] A generation network layer is used to obtain a homography matrix between the input image and the generated image and a corresponding score through regression prediction according to the feature vector and the generated image.

[0027] Optionally, in the above monocular SLAM initialization method, the step of obtaining a training data set includes:

[0028] Acquire a plurality of key frame images, fourth feature points of the plurality of key frame images, and second map points corresponding to the plurality of key frame images;

[0029] Calculating the observation weight of the second map point according to the multiple key frame images;

[0030] For each key frame image, determining a common view frame image of the key frame image and a fifth feature point of the common view frame image;

[0031] Obtaining a basic matrix between the key frame image and the common view frame image according to the camera pose of the key frame image and the camera pose of the common view frame image;

[0032] Obtaining a score of the key frame image according to the observation weight, the basic matrix, and a matching relationship between the fourth feature point and the fifth feature point of the key frame image;

[0033] A training data set is obtained according to the multiple key frame images, the common view frame images of the multiple key frame images, the second homography matrix of the multiple key frame images and the scores of the multiple key frame images, wherein the second homography matrix is ​​converted from the basic matrix.

[0034] Optionally, in the above monocular SLAM initialization method, the step of obtaining the score of the key frame image according to the observation weight, the basic matrix, and the matching relationship between the fourth feature point and the fifth feature point of the key frame image includes:

[0035] Calculating the epipolar line of the fifth feature point according to the basic matrix and the fourth feature point of the key frame image;

[0036] Calculating a matching score between a fourth feature point and the fifth feature point of the key frame image according to a distance between the fifth feature point and the epipolar line;

[0037] The score of the key frame image is obtained according to the observation weight and the matching score.

[0038] Optionally, in the above monocular SLAM initialization method, the step of obtaining a training data set according to the multiple key frame images, the common view frame images of the multiple key frame images, the second homography matrix of the multiple key frame images, and the scores of the multiple key frame images includes:

[0039] Filtering the multiple key frame images and the common view frame images of the multiple key frame images according to the scores of the multiple key frame images to obtain filtered key frame images and common view frame images;

[0040] sorting the screened common view frame images according to the scores of the screened key frame images to obtain sorted common view frame images;

[0041] Calculating the angle value of the sorted common view frame image according to the rotation matrix between the filtered key frame image and its corresponding filtered common view frame image;

[0042] According to the angle value, inserting the sorted common view frame image into a rotation histogram to obtain a count value of the rotation histogram;

[0043] According to the count value, a final key frame image and its corresponding common view frame image, a second homography matrix and a score are determined to obtain a training data set.

[0044] In a second aspect, the present invention provides a monocular SLAM initialization device, the device comprising:

[0045] A data acquisition module, used to acquire a first frame image and a first feature point of the first frame image;

[0046] An image processing module, used for inputting the first frame image into a pre-trained generative network model to obtain a first homography matrix; wherein the generative network model is constructed based on a convolutional neural network and a generative adversarial network;

[0047] A perspective transformation module, used for performing perspective transformation on the first frame image according to the first homography matrix to obtain a virtual frame image;

[0048] A feature conversion module, used to convert the first feature point to the virtual frame image using the first homography matrix to obtain a second feature point of the virtual frame image;

[0049] An initialization module is used to obtain a first map point using a triangulation algorithm according to the second feature point and the corresponding first feature point.

[0050] In a third aspect, the present invention provides a monocular SLAM initialization device, the device comprising a processor and a memory, the memory storing a monocular SLAM initialization program, and when the monocular SLAM initialization program is executed by the processor, the monocular SLAM initialization method as described above is implemented.

[0051] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by one or more processors, the monocular SLAM initialization method as described above is implemented.

[0052] The above one or more technical solutions provided by the present invention may have the following advantages or at least achieve the following technical effects:

[0053] The present invention proposes a monocular SLAM initialization method, device, equipment and storage medium. The method inputs a first frame image into a pre-trained generative network model to obtain a first homography matrix, performs perspective transformation on the first frame image, and obtains a virtual frame image; then uses the first homography matrix to convert the first feature point to the virtual frame image to obtain the second feature point of the virtual frame image; then uses a triangulation algorithm to obtain the first map point based on the second feature point and the first feature point in the corresponding first frame image, thereby realizing the initialization of the monocular SLAM. Compared with the existing method that requires two video frames with large parallax to initialize, the present invention only requires one video frame image to realize monocular SLAM initialization, which is simple for user operation and has low requirements for user operation; by combining the convolutional neural network with the generative adversarial network to obtain a generative network model, a virtual frame image is generated for initialization, and initialization can be completed according to the first frame image during application startup, so as to complete initialization as early as possible and quickly start the monocular SLAM system and its related applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0055] Figure 1 It is a schematic diagram of the process of the first embodiment of the monocular SLAM initialization method of the present invention;

[0056] Figure 2 A schematic diagram of the hardware structure of a monocular SLAM initialization device according to the present invention;

[0057] Figure 3It is a schematic diagram of the process of the second embodiment of the monocular SLAM initialization method of the present invention;

[0058] Figure 4 It is a schematic diagram of the epipolar geometry of the feature points of the key frame image and the common view frame image in the second embodiment of the monocular SLAM initialization method of the present invention;

[0059] Figure 5 Schematic diagram of the structure of the hybrid neural network in step A200 of the second embodiment of the monocular SLAM initialization method of the present invention;

[0060] Figure 6 Schematic diagram of functional modules of the first embodiment of the monocular SLAM initialization device of the present invention.

[0061] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] It should be noted that in the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the existence of other identical elements in the process, method, article or system including the element. In addition, in the present invention, if there is a description involving "first", "second", etc., the description of "first", "second", etc. is only for descriptive purposes, and cannot be understood as indicating or implying its relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the present invention, the use of suffixes such as "module", "component" or "unit" used to represent elements is only to facilitate the description of the present invention, and has no specific meaning in itself. Therefore, "module", "component" or "unit" can be used in a mixed manner. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, the technical solutions of various embodiments can be combined with each other, but this is based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0064] Analysis of the prior art found that in the current monocular SLAM system, generally by selecting two key video frame images with large parallax, solving the homography matrix or basic matrix based on the epipolar geometry method, selecting one of them to restore the camera pose, and generating map points by triangulation, the initialization operation of the monocular SLAM system is realized. The problem with this monocular SLAM initialization method is that two video frames with large parallax are required for initialization, and the success of initialization depends largely on the actual shooting method of the user. In some pure rotation, too close field of view or too slow movement, it is difficult for monocular SLAM to initialize successfully, and there is a problem of low success rate of monocular SLAM initialization. In addition, it is difficult for new users to master the correct camera method to guide initialization, and it is easy to fail to initialize, and there is a problem of difficult operation, resulting in a poor user experience. In actual application, it is often necessary to guide users to shoot in a recommended way through some interactive texts and animations to improve the success rate of monocular SLAM initialization, which in turn increases the complexity of the system.

[0065] In view of the technical problem that the success rate of monocular SLAM initialization is low in the prior art, the present invention provides a monocular SLAM initialization method, and the overall idea is as follows:

[0066] Acquire a first frame image and a first feature point of the first frame image; input the first frame image into a pre-trained generative network model to obtain a first homography matrix; wherein the generative network model is constructed based on a convolutional neural network and a generative adversarial network; perform a perspective transformation on the first frame image according to the first homography matrix to obtain a virtual frame image; use the first homography matrix to convert the first feature point to the virtual frame image to obtain a second feature point of the virtual frame image; and use a triangulation algorithm to obtain a first map point according to the second feature point and the corresponding first feature point.

[0067] Through the above technical solution, the initialization of monocular SLAM is realized. Compared with the existing method that requires two video frames with large parallax for initialization, monocular SLAM initialization can be realized with only one video frame image, which is simple for user operation and has low requirements for user operation; by combining the convolutional neural network with the generative adversarial network to train the generative network model, a virtual frame image is generated for initialization, and the initialization can be completed according to the first frame image during the application startup, so as to complete the initialization as early as possible and quickly start the monocular SLAM system and its related applications.

[0068] The monocular SLAM initialization method, device, equipment and storage medium provided by the present invention are described in detail below with reference to the accompanying drawings through specific embodiments and implementation modes.

[0069] Embodiment 1

[0070] Reference Figure 1 A flow chart of the present invention is provided, and a first embodiment of the monocular SLAM initialization method of the present invention is proposed. The monocular SLAM initialization method can be applied to a monocular SLAM initialization device, and specifically can be applied to the AR underlying technology of an application program in the device.

[0071] The monocular SLAM initialization device refers to a terminal device, network device or system device that can achieve network connection. For example, it can be a terminal device such as a mobile phone, computer, tablet computer, embedded industrial computer, or a network device such as a server or cloud platform. It can also be a system device that includes terminal devices such as computers and mobile phones and network devices such as edge servers and cloud servers.

[0072] like Figure 2 , which is a schematic diagram of the hardware structure of a monocular SLAM initialization device. The monocular SLAM initialization device may include: a processor 1001, such as a CPU (Central Processing Unit), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005.

[0073] Those skilled in the art will understand that Figure 2 The hardware structure shown in the figure does not constitute a limitation on the monocular SLAM initialization device of the present invention, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0074] Specifically, the communication bus 1002 is used to realize the connection and communication between these components;

[0075] The user interface 1003 is used to connect to the client and perform data communication with the client. The user interface 1003 may include an output unit, such as a display screen, and an input unit, such as a keyboard;

[0076] The network interface 1004 is used to connect to the backend server and perform data communication with the backend server. The network interface 1004 may include an input / output interface, such as a standard wired interface, a wireless interface, such as a Wi-Fi interface;

[0077] The memory 1005 is used to store various types of data, which may include, for example, instructions for any application or method in the monocular SLAM initialization device, and data related to the application. The memory 1005 may be a high-speed RAM memory or a stable memory, such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the processor 1001, and further reference is made to Figure 2 , the memory 1005 may include an operating system, a network communication module, a user interface module, and a monocular SLAM initialization program;

[0078] The processor 1001 is used to call the monocular SLAM initialization program stored in the memory 1005, and perform the following operations:

[0079] Acquire a first frame image and a first feature point of the first frame image;

[0080] Inputting the first frame image into a pre-trained generative network model to obtain a first homography matrix; wherein the generative network model is constructed based on a convolutional neural network and a generative adversarial network;

[0081] According to the first homography matrix, perspective transformation is performed on the first frame image to obtain a virtual frame image;

[0082] Using the first homography matrix, converting the first feature point to the virtual frame image to obtain the second feature point of the virtual frame image;

[0083] A first map point is obtained by using a triangulation algorithm according to the second feature point and the corresponding first feature point.

[0084] Based on the above monocular SLAM initialization device, the following is combined Figure 1The flow chart shown in FIG. 1 is a flow chart that describes the monocular SLAM initialization method of this embodiment in detail. The method may include the following steps:

[0085] Step S100: Acquire a first frame image and first feature points of the first frame image.

[0086] Specifically, the monocular SLAM system set on the monocular SLAM initialization device may include an image acquisition device such as a camera, a camera, etc. When the monocular SLAM system is started and the image acquisition device is turned on, the image of the first frame is acquired and determined as the first frame image. Then the first frame image can be processed separately, including extracting feature points on the first frame image to obtain the first feature points, and generating the first homography matrix according to the first frame image and the generation network model. Among them, the feature point refers to a two-dimensional point on the image, generally referring to a point with obvious features such as a corner point.

[0087] In this embodiment, the monocular SLAM initialization device in which the method is applied to the system device structure is used as an example for explanation, so that some steps of the method can be executed on terminal devices such as mobile phones and computers, and some steps can be executed on network devices such as cloud servers and edge servers. Specifically, the terminal device collects the first frame image F1, and then opens two threads: Thread 1 sends the first frame image F1 to the server, and the image can also be compressed when sending it, so as to send it to the server as soon as possible, and the server independently processes the first frame image F1; Thread 2 is executed locally on the terminal device, and features are extracted from the first frame image F1. ORB (Oriented FAST and Rotated BRIEF, feature point detection and extraction algorithm) can be used for feature extraction, and 1000 ORB feature points are extracted to obtain the first feature point of the first frame image F1. Among them, the number of extracted feature points can be set according to the actual situation, and Thread 1 and Thread 2 can be performed simultaneously or in sequence.

[0088] Step S200: inputting the first frame image into a pre-trained generative network model to obtain a first homography matrix; wherein the generative network model is constructed based on a convolutional neural network and a generative adversarial network.

[0089] Specifically, the generative network model is constructed and trained based on convolutional neural networks (CNN) and generative adversarial networks (GAN). After obtaining the first frame of image, the monocular SLAM initialization device can continue to input the first frame of image into the pre-trained generative network model, and the model directly outputs the first homography matrix after operation. Among them, the specific operation process of the generative network model can be: feature extraction of the input first frame of image, output of feature vector; then according to the feature vector, the generated image of the first frame of image is obtained by upsampling; then according to the feature vector and the generated image, the homography matrix between the first frame of image and the corresponding generated image, that is, the first homography matrix, is obtained by regression prediction. Among them, the number of generated images can be multiple, so as to obtain multiple first homography matrices correspondingly, and optionally, the scores corresponding to the multiple first homography matrices can also be obtained, so as to sort or screen the multiple first homography matrices to obtain the most accurate homography matrix. The architecture of the generative network model is different, and the operation process is different. The specific architecture and operation process can also be set according to actual needs.

[0090] In this embodiment, after the cloud server receives the first frame image F1 sent by the terminal device, it extracts features and upsamples the first frame image F1 to obtain three frames of generated images F2, and then obtains the corresponding three first homography matrices H1 through regression prediction, and then returns the three first homography matrices H1 to the terminal device. It should be noted that the advantage of setting the calculation steps of the model in the cloud in this embodiment is that it can save the computing power of the terminal device, and the cloud server can provide greater computing power to ensure that there is enough space for operations such as training, calculation and optimization of the generated network model. It can also reduce the requirements of the terminal device, and does not require it to have a high computing power, and ensure that the thread two running locally can be carried out simultaneously, saving the running time of the method and completing the initialization operation as soon as possible.

[0091] Step S300: performing perspective transformation on the first frame image according to the first homography matrix to obtain a virtual frame image.

[0092] Specifically, perspective transformation refers to a transformation that utilizes the condition that the perspective center, the image point, and the target point are collinear, and according to the law of perspective rotation, the perspective plane is rotated around the perspective axis by a certain angle, thereby destroying the original projection light beam, and still maintaining the projection geometry on the perspective plane unchanged. Here, according to the condition that the perspective center, the first frame image, and the virtual frame image obtained after the transformation are collinear, the first frame image is perspectively transformed according to the first homography matrix to obtain a virtual frame image corresponding to the first frame image. Optionally, when there are multiple first homography matrices obtained with corresponding scores, the first frame image can be perspectively transformed with the first homography matrix in turn according to the scores, so that multiple virtual frame images with the same number as the first homography matrix can be obtained.

[0093] In this embodiment, after the terminal device receives the three first homography matrices H1 sent by the cloud server, three virtual frame images VF are obtained through perspective transformation according to the first frame image F1 and the first homography matrix H1.

[0094] Step S400: using the first homography matrix, converting the first feature points to the virtual frame image to obtain second feature points of the virtual frame image.

[0095] Specifically, after obtaining the first feature point of the first frame image, the first homography matrix and the virtual frame image of the first frame image, the first feature point is converted to the virtual frame image according to the first homography matrix to obtain the feature point on the virtual frame image, that is, the second feature point. Optionally, during the conversion, the feature points that exceed the boundary of the virtual frame image can be eliminated; the feature points can also be screened according to the response value, for example, according to the feature points that do not exceed the boundary after conversion to the virtual frame image, the response values ​​corresponding to these feature points are counted to obtain a maximum response value, and then the feature points whose response values ​​are less than the preset response value are eliminated, and finally the second feature point of the virtual frame image is obtained, wherein the preset response value can be set directly or according to the maximum response value, for example, it is set to the product of the maximum response value and the set coefficient. Cleaning up the feature points on the virtual frame image, retaining more effective feature points, and determining them as the second feature points of the final virtual frame image can improve the accuracy of the constructed first map points and ensure the accuracy of the initialization operation.

[0096] In this embodiment, according to the three first homography matrices H1, the following operations are performed respectively: 1000 feature points in the first frame image F1 are converted to each virtual frame image VF, and feature points beyond the boundary of the virtual frame image VF are eliminated, and based on the maximum response value of the feature points within the boundary, feature points with response values ​​less than the preset response value are eliminated to obtain the feature points finally retained on the virtual frame image VF, that is, the second feature points are obtained, and the preset response value here can be set to preset response value = 0.8*maximum response value. After completing the above operations, the second feature points of the three virtual frame images VF can be obtained respectively, and the number of second feature points is less than or equal to 1000. Assuming that each virtual frame image VF has 800 second feature points here, there are 2400 second feature points in total.

[0097] Step S500: obtaining a first map point using a triangulation algorithm according to the second feature point and the corresponding first feature point.

[0098] Specifically, since the second feature point is obtained by conversion based on the first feature point or by screening after conversion, each second feature point will have a first feature point associated with it, but not every first feature point has a corresponding second feature point. Therefore, here we can first determine the first feature point corresponding to the second feature point, and then generate 3D map points based on the second feature point and the first feature point corresponding to it using a triangulation algorithm, and count the first feature points corresponding to the second feature points of multiple virtual frame images. In the same way, we can obtain multiple 3D map point sets, and after summarizing, we can obtain the first map point. The 3D map points here are essentially obtained based on the first frame image and the virtual frame image, and can also be called virtual map points. These map points are three-dimensional points for modeling the target scene targeted by the monocular SLAM system.

[0099] Optionally, the camera pose of the first frame image can be obtained by minimizing the reprojection error based on the matching relationship between each virtual map point in the first map point and the first feature point. At this point, the single frame initialization process of the monocular SLAM system ends. Other corresponding operations can be performed based on the first map point and the camera pose.

[0100] In this embodiment, after finding the second feature point and the corresponding first feature point according to the correlation between the first feature point on the first frame image F1 and the second feature point on the virtual frame image VF, each second feature point and its corresponding first feature point can be regarded as a pair of matching point pairs. In the optional scheme, after cleaning the feature points on the virtual frame image VF, multiple pairs of matching point pairs after cleaning can be obtained. In this embodiment, based on the three virtual frame images VF after cleaning the feature points, three matching point pair sets can be obtained, each matching point pair set contains 800 second feature points and their corresponding 800 first feature points, that is, there are 800 pairs of matching point pairs; then, based on these 800 pairs of matching point pairs, a triangulation algorithm can be used to generate 3D map points, and 800 virtual map points generated by the first frame image F1 and a virtual frame image VF will be obtained. In actual implementation, the number of virtual map points may be reduced because a matching point pair may fail to generate 3D map points. For the three virtual frame images VF, a large number of virtual map points generated by the first frame image F1 and the three virtual frame images VF will be obtained in the same way. Assuming that three groups of 800 virtual map points can be obtained here, that is, 2400 virtual map points can be obtained. The first map point can be obtained by summarizing these virtual map points. Optionally, the camera pose of the first frame image F1 can be further obtained, thereby realizing the single-frame initialization operation of the monocular SLAM system.

[0101] The monocular SLAM initialization method provided in this embodiment obtains the first homography matrix by inputting the first frame image into the pre-trained generative network model, performs perspective transformation on the first frame image, and obtains the virtual frame image; then uses the first homography matrix to convert the first feature point to the virtual frame image to obtain the second feature point of the virtual frame image; then uses the triangulation algorithm to obtain the first map point based on the second feature point and the first feature point in the corresponding first frame image, thereby realizing the initialization of the monocular SLAM. Compared with the existing method that requires two video frames with large parallax to initialize, the present invention only needs one video frame image to realize the monocular SLAM initialization, which is simple for user operation and has low requirements for user operation; by combining the convolutional neural network with the generative adversarial network to obtain the generative network model, a virtual frame image is generated for initialization, and the initialization can be completed according to the first frame image during the application startup, so as to complete the initialization as early as possible and quickly start the monocular SLAM system and its related applications.

[0102] Embodiment 2

[0103] Based on the same inventive concept, Figures 3 to 5 Based on the first embodiment, a second embodiment of the monocular SLAM initialization method of the present invention is proposed. The method can also be applied to the monocular SLAM initialization device in the first embodiment.

[0104] Combine the following Figure 3 The flow chart shown in FIG. 1 is a flow chart that describes the monocular SLAM initialization method of this embodiment in detail.

[0105] After the step of obtaining the first map point by using a triangulation algorithm based on the second feature point and the corresponding first feature point, the following steps S600-S910 can be executed in sequence to achieve the purpose of further performing a second frame transition after the single-frame initialization operation to obtain a more accurate 3D map point; steps S100-S500 can also be independently executed steps to achieve the purpose of quickly completing the initialization operation.

[0106] Furthermore, after step S500, the method may further include:

[0107] Step S600: Acquire a second frame image and a third feature point of the second frame image.

[0108] Specifically, after the monocular SLAM initialization device completes steps S100-S500, the initialization operation has been completed, but the first map point obtained may not be very accurate. Therefore, a map optimization step can be added to transition with the second frame image to obtain a more accurate map point. The overall process of the monocular SLAM system includes three stages: initialization, transition, and normal operation. The second frame image can be the image of the second frame actually collected by the image acquisition device, or it can be the image of the Nth frame after the first frame image. After obtaining the second frame image, the feature points on it will also be extracted to obtain the third feature points.

[0109] In this embodiment, after the terminal device captures the second frame image F2, there is no need to start thread one again. Instead, thread two continues to be executed locally on the terminal device, and ORB is used to perform feature extraction on the second frame image F2 to extract 1000 ORB feature points, thereby obtaining 1000 third feature points of the second frame image F2.

[0110] Step S700: taking the first frame image as a reference frame, performing feature point matching on the first frame image and the second frame image to obtain first feature points that are successfully matched.

[0111] Specifically, the first frame image is used as a reference frame, and feature point matching is performed on the first frame image and the second frame image to obtain first feature points that are successfully matched. At this time, the number of first feature points that are successfully matched is less than or equal to the number of first feature points. Specifically, the first frame image is used as a reference frame, and each first feature point is matched with each third feature point. If there is a third feature point corresponding to the first feature point on the second frame image, the first feature point is determined as a first feature point that is successfully matched; if there is no third feature point corresponding to the first feature point on the second frame image, the first feature point is determined as a first feature point that is unmatched; if these first feature points that are unmatched have corresponding virtual map points in the first map points, these virtual map points can be directly deleted.

[0112] In this embodiment, the first frame image F1 is used as a reference frame, and the 1000 first feature points on the first frame image F1 and the 1000 third feature points on the second frame image F2 are matched to obtain successfully matched feature point pairs consisting of the first feature points and the third feature points; then the first feature point in these feature point pairs is determined as the successfully matched first feature point. Among them, the number of feature point pairs is less than or equal to 1000, and the number of successfully matched first feature points is also less than or equal to 1000. In this embodiment, it is assumed that 900 pairs of feature point pairs are successfully matched, and 900 successfully matched first feature points can be obtained.

[0113] Step S800: using the first homography matrix, projecting the first map point corresponding to the successfully matched first feature point onto the second frame image to obtain the projection point in the second frame image.

[0114] Specifically, it is determined whether there is a virtual map point corresponding to the first feature point successfully matched in the first map point, and a number of virtual map points can be obtained, and the number of the virtual map points is less than or equal to the number of the first feature points successfully matched. Then, using the first homography matrix, the several virtual map points are projected onto the second frame image, and the projection points in the second frame image can be obtained. At this time, there is a matching relationship between the first feature point successfully matched, the third feature point associated with the first feature point successfully matched, and the projection point. It should be noted that for the case where a first homography matrix and a virtual frame image are obtained in the aforementioned steps, since not all first feature points have corresponding virtual map points, the first feature points successfully matched here may not all have corresponding virtual map points, so that the number of virtual map points corresponding to the first feature points successfully matched may be less than or equal to its own number, or even less than the number of second feature points on the aforementioned virtual frame image.

[0115] In this embodiment, it is determined whether there are virtual map points corresponding to the 900 successfully matched first feature points in the first map points. Assuming that 720 first map points can be obtained, then, using the first homography matrix, these 720 first map points are projected onto the second frame image F2 respectively, and 720 projection points in the second frame image F2 are obtained. In view of the situation in which the first map points consisting of 2400 virtual map points are obtained according to the three first homography matrices and the three virtual frame images VF in the aforementioned steps, it can be assumed here that 720*3=2160 first map points are obtained. Then, using the first homography matrix, these 2160 first map points are projected onto the second frame image F2, and 2160 projection points can be obtained in the second frame image. The remaining 2400-2160=240 virtual map points in the first map points can be directly deleted.

[0116] Step S900: Determine the first map point corresponding to the projection point that meets a preset condition as a final map point, wherein the preset condition is that the third feature point exists within a preset range centered on the projection point.

[0117] Specifically, in the second frame image, within the preset range centered on the projection point, search whether there is a third feature point, or whether there is a third feature point belonging to the aforementioned feature point pair, that is, a third feature point that is successfully matched; if there are multiple third feature points, the third feature point with the shortest distance to the projection point is used as the matching point, if there is a third feature point, the third feature point is directly used as the matching point, and then the virtual map point corresponding to the projection point with the matching point is retained; if there is no third feature point, the virtual map point corresponding to the projection point can be deleted. According to the above method, each projection point is traversed to obtain the final 3D map point that meets the conditions, that is, the final map point. At this point, the transition process based on the second frame image, that is, the process of optimizing the map point, ends, and the frame images subsequently obtained by the monocular SLAM system can perform other corresponding operations based on the final map point. Among them, the preset range can be a circular range with the projection point as the center and the preset value as the radius.

[0118] In this embodiment, based on each projection point PP of the 2160 projection points, within the range of a circle with PP as the center and a radius r, where r can be set according to actual conditions, the third feature point is searched to delete the virtual map point corresponding to the projection point PP in the first map point where the third feature point does not exist, and retain the virtual map point corresponding to the projection point PP in the first map point where the matching point exists. After the above operation is performed on all projection points, the remaining first map points can be obtained, and the number of remaining first map points here is less than or equal to 2160. Based on the aforementioned deletion of other virtual map points other than the 2160 virtual map points in the first map point, the 3D map points in the remaining first map point here are all virtual map points corresponding to the projection point PP with a matching point, that is, the final map point is obtained.

[0119] Optionally, after step S900, the method may further include:

[0120] Step S910: According to the correspondence between the final map point and the third feature point, the camera pose of the second frame image is obtained by minimizing the reprojection error method.

[0121] Specifically, according to the association relationship between the virtual map point in the final map point obtained in step S900 and the first feature point that matches successfully and the third feature point that matches successfully, the camera pose of the second frame image is obtained by minimizing the reprojection error method. At this point, the single-frame initialization process and the second-frame transition process of the monocular SLAM system are all completed, and then starting from the third frame image, the subsequent corresponding operations can be completed using the conventional SLAM tracking algorithm.

[0122] The monocular SLAM initialization method provided in this embodiment performs feature point matching on a first frame image and a second frame image, projects a first map point corresponding to a successfully matched first feature point onto the second frame image, obtains a projection point in the second frame image, and then uses a projection point having a third feature point within a preset range centered on the projection point as a projection point that meets preset conditions, so as to determine the first map point corresponding to the projection point that meets the conditions as the final map point, and continues to execute the second frame transition phase after the initialization phase, thereby achieving the purpose of optimizing 3D map points according to the second frame image, and ensuring that the processing of subsequent frame images can be more accurate.

[0123] The following steps A100-A300 can be independently executed step branches to achieve the purpose of obtaining the generated network model, or the method can be executed in the order of A100-A300, S100-S900 to achieve the purpose of initializing the generated network model after pre-acquiring the generated network model.

[0124] Furthermore, before step S200, the method may further include:

[0125] Step A100: Obtain a training data set;

[0126] Specifically, step A100 may include:

[0127] Step A110: Acquire a plurality of key frame images, fourth feature points of the plurality of key frame images, and second map points corresponding to the plurality of key frame images.

[0128] Specifically, a key frame image refers to an experimental image containing sufficiently rich feature information. A large number of key frame images can be obtained as training data, and then several key frame images are extracted from them, and feature extraction is performed to obtain the fourth feature point of the key frame image. Then, a 3D map point is generated for the key frame image and its fourth feature point. For example, by selecting two key frame images with large parallax, solving the homography matrix based on the epipolar geometry method, and then using triangulation to generate 3D map points to obtain the second map point. A large number of key frame images can be selected from a large number of images of the target scene specifically applied to the monocular SLAM system. The number of key frame images can be set according to actual conditions, and the number of key frame images required to generate the second map point can also be selected according to actual conditions.

[0129] For example, several key frame images with large parallax are obtained, and 100 two-dimensional feature points of each key frame image are extracted as the fourth feature points; then the fourth feature points of these key frame images are matched, and 3D map points are generated based on the successfully matched point pairs to obtain the second map point. Among them, there may be requirements for the number of point pairs. If the requirements are not met, the key frame images can be replaced and two key frame images that meet the requirements can be found again. Assuming that the number of matching point pairs obtained here meets the requirements, such as more than 20 point pairs are successfully matched, 20 3D map points can be generated correspondingly based on these 20 point pairs to obtain the second map point.

[0130] Step A120: Calculate the observation weight of the second map point according to the multiple key frame images.

[0131] Specifically, for each map point in the second map point, the number of observations is counted, that is, the number of times this map point appears in different key frame images, to obtain the observation weight. The calculation formula of the observation weight is:

[0132]

[0133] Where i represents the sequence number of the map point in the second map point, W i Represents the i-th map point MP i The observation weight, OB i Represents the i-th map point MP iThe number of observations of , max(OB) represents the maximum value of the number of observations of all map points in the second map point.

[0134] According to the above calculation formula, the observation weight W of each map point on the second map point can be calculated i .

[0135] Step A130: for each key frame image, determine the common view frame image of the key frame image and the fifth feature point of the common view frame image.

[0136] Specifically, for each key frame image, the common view frame image of the key frame image is determined, and one or several appropriate common view frame images can be selected. When determining the common view frame image, it can be screened according to the degree of common view between the key frame image and the common view frame image. After obtaining the common view frame image, the feature points of each common view frame image can also be extracted to obtain the fifth feature point.

[0137] For example, for a certain key frame image KF1, the first m common view frame images with the highest common view degree can be taken out, that is, there are m best common view frame images, and subsequent operations are performed on the key frame image KF1 and each common view frame image. It is assumed here that the common view frame image currently taken out is KF2.

[0138] Step A140: Obtain a basic matrix between the key frame image and the common view frame image according to the camera pose of the key frame image and the camera pose of the common view frame image.

[0139] Specifically, according to the camera pose of the key frame image and the camera pose of the common view frame image, the basic matrix between the key frame image and the common view frame image is calculated, wherein the calculation formula for calculating the basic matrix is:

[0140]

[0141] Among them, F 12 represents the basic matrix between the key frame image KF1 and the common view frame image KF2;

[0142] K represents the intrinsic parameter matrix of the camera;

[0143] (R, t) represents the external parameter matrix of the camera, (R 12 , t 12 ) represents the transformation matrix from the key frame image KF1 to the common view frame image KF2, where R 12 =R 1w *R 2w T , t 12 =-R 1w *R 2w T *t 2w+t 1w , (R 1w , t 1w ) represents the camera pose of the key frame image KF1, (R 2w , t 2w ) represents the camera pose of the common viewing frame image KF2;

[0144] t^ 12 Indicates t 12 The antisymmetric matrix of .

[0145] Step A150: obtaining a score of the key frame image according to the observation weight, the basic matrix, and a matching relationship between the fourth feature point and the fifth feature point of the key frame image;

[0146] More specifically, step A150 may include:

[0147] Step A151: calculating the epipolar line of the fifth feature point according to the basic matrix and the fourth feature point of the key frame image;

[0148] Step A152: calculating a matching score between the fourth feature point of the key frame image and the fifth feature point according to the distance between the fifth feature point and the epipolar line;

[0149] Step A153: Obtain the score of the key frame image according to the observation weight and the matching score.

[0150] like Figure 4 The figure shows the epipolar geometry diagram of the feature points of the key frame image and the common view frame image. In the figure, the optical center of the key frame image KF1 is O1, and the optical center of the common view frame image KF2 is O2. It is assumed that the fourth feature point x1 of the key frame image KF1 and the fifth feature point x2 of the common view frame image KF2 are a pair of feature points, the coordinates of the fourth feature point x1 are P1, and the coordinates of the fifth feature point x2 are P2.

[0151] According to the fundamental matrix F 12 and the coordinates P1 of the fourth feature point x1, we can calculate the epipolar line l2 (i.e., ray O2x2 in the figure) where the fifth feature point x2 is located. Assuming that the equation of the epipolar line l2 is ax+by+c=0, then,

[0152]

[0153] Thus the polar line l2 is calculated;

[0154] Then, the distance dist between the fifth feature point x2 and the epipolar line l2 can be calculated:

[0155]

[0156] Thus, the matching score N between the fourth feature point x1 and the fifth feature point x2 is calculated. match :

[0157]

[0158] Among them, scale represents the scaling ratio of the pyramid level where the feature point is located, and 3.84 is the chi-square distribution value with a degree of freedom of 1;

[0159] Finally, the score of the key frame image is calculated based on the observation weight of the second map point corresponding to the feature point pair formed by the fourth feature point and the fifth feature point, and the matching score of this feature point pair:

[0160]

[0161] Wherein, n represents the number of matching point pairs formed by the fifth feature point and the fourth feature point of the common viewing frame image KF2, W j Nmatch represents the observation weight corresponding to the map point of the jth matching point pair. j represents the matching score of the jth matching point pair, λ represents an adjustable parameter that can be set according to actual needs, and t cj represents the disparity between the key frame image KF1 and the i-th co-viewing frame image, and the calculation method is the same as the above t 12 The calculation method is the same as that of , and will not be repeated here.

[0162] Optionally, after calculating the scores of each key frame image in the above manner, each key frame and its corresponding common view frame image, homography matrix and score can be recorded and stored in the form of a mapping table, wherein the basic matrix and the homography matrix can be converted into each other because they only differ by one camera intrinsic parameter. Therefore, the homography matrix can be obtained based on the basic matrix calculated above.

[0163] Step A160: Obtain a training data set based on the multiple key frame images, the common view frame images of the multiple key frame images, the second homography matrix of the multiple key frame images and the scores of the multiple key frame images, wherein the second homography matrix is ​​converted from the basic matrix.

[0164] In order to achieve better model training results, different from the conventional random shuffling of data, a reasonable data batching rule can be designed to obtain the final training data set.

[0165] More specifically, step A160 may include:

[0166] Step A161: screening the multiple key frame images and the common view frame images of the multiple key frame images according to the scores of the multiple key frame images to obtain screened key frame images and common view frame images.

[0167] After calculating the scores of all key frame images according to step A150, the selected key frame images and several common view frame images of each key frame image are screened according to the scores, and the screened key frame images and common view frame images can be obtained. For example, for each key frame image, its corresponding 15 common view frame images are taken out, and 15 scores are calculated according to the key frame image and each common view frame image according to the above method, and the 15 common view frame images are traversed to compare and obtain a highest score; then, the key frame images with scores lower than the set value are deleted, that is, the key frame images with scores higher than the set value are retained, wherein the set value can be the product of the highest score and the set coefficient, such as the set value = the highest score * 0.75. In this way, training data consisting of key frame images with higher overall quality can be obtained.

[0168] Step A162: sorting the screened common view frame images according to the scores of the screened key frame images to obtain sorted common view frame images;

[0169] Step A163: calculating the angle value of the sorted common view frame image according to the rotation matrix between the filtered key frame image and its corresponding filtered common view frame image;

[0170] Step A164: inserting the sorted common view frame image into a rotation histogram according to the angle value to obtain a count value of the rotation histogram;

[0171] Step A165: According to the count value, determine the final key frame image and its corresponding common view frame image, the second homography matrix and the score to obtain a training data set.

[0172] Based on the filtered key frame images, for each key frame image, take out its rotation matrix R with the corresponding common view frame image 12, and convert it into Euler angles for easy comparison. Set a rotation histogram, with 30° as an angle interval. The angle of the angle interval can be set according to the actual situation. Sort the filtered common view frame images obtained in step A161 according to the scores from high to low to obtain the sorted common view frame images; then insert the sorted common view frame images into the training data set one by one, and each time a data is inserted, take the average as an angle value according to the Euler angle value, count it in the corresponding rotation histogram, and obtain a count value. When there is a rotation histogram count value in a certain angle interval that is greater than one times the average, retain the data inserted into the angle interval to make the angle distribution of the training data more even. Among them, when determining the requirements of the final key frame image and its corresponding common view frame image according to the count value, the multiple or difference relationship can be set according to the actual situation, so as to select the final training data to be retained according to the actual situation. Then, the final key frame image, the common view frame image, the key frame image and the second homography matrix of the common view frame image, and the score of the key frame image are used as the training data set. Among them, the second homography matrix is ​​obtained based on the transformation of the basic matrix of the key frame image and the common view frame image.

[0173] Step A200: constructing a hybrid neural network;

[0174] Wherein, the hybrid neural network comprises:

[0175] The feature extraction layer is used to extract features from the input image and output feature vectors;

[0176] A convolutional network layer, used for obtaining a generated image of the input image by upsampling according to the feature vector, the convolutional network layer comprising a convolution module, an activation module and a custom module;

[0177] A generation network layer is used to obtain a homography matrix between the input image and the generated image and a corresponding score through regression prediction according to the feature vector and the generated image.

[0178] Specifically, Figure 5 The figure shows a schematic diagram of the structure of the hybrid neural network. The hybrid neural network combines the CNN network and the GAN network. Unlike the traditional GAN ​​network, which uses the original input image as the supervision target, the score in the training data set obtained in step A100 is used as the supervision target.

[0179] The hybrid neural network adopts a multi-level fusion method. The feature extraction layer can extract features from the input image and output feature vectors. The upper layer of the convolutional network layer adopts the U-Net network, and the lower layer adopts the CNN network. In the upsampling stage, the feature map of the input image can be obtained by upsampling according to the feature vector. These feature maps can be feature maps of different sizes. Then, in the downsampling stage, these feature maps are embedded in the CNN network. The CNN network adopts an adaptive attention mechanism, which is composed of a 3x3Conv convolution module, a ReLU activation module, and an AAD custom module. Among them, the AAD custom module is used to integrate the feature map from the upper U-Net network and the feature map output by the ReLU activation module. The generation network layer is the last layer of the hybrid neural network, regressing a 10x1 vector, in which the first dimension is the score, and the remaining 9 scalars are the corresponding homography matrix values. So far, we have obtained a hybrid neural network that can receive a key frame image, generate the corresponding common view frame image, and obtain the corresponding homography matrix and score.

[0180] Step A300: training the hybrid neural network according to the training data set to obtain the generated network model.

[0181] Specifically, for the key frame image in the training data set, a hybrid neural network is input, a virtual common view frame image is output, and after obtaining the corresponding homography matrix and score, the output result is compared with the common view frame image, the second homography matrix and the score stored in the training data set to implement the training of the hybrid neural network. The network converges to obtain an optimal hybrid neural network and its parameters, and thus a pre-trained model is obtained, namely the generative network model.

[0182] For more implementation details in the specific implementation of the above method steps, please refer to the description of the specific implementation in Example 1. For the sake of brevity of the description, they will not be repeated here.

[0183] The monocular SLAM initialization method provided in this embodiment proposes a method of screening training data by scores in the preparation of training data for generating a network model, thereby improving the accuracy of training data; and proposes a hybrid neural network combining a U-Net network and a CNN network. The generated network model obtained by training directly generates a homography matrix in an end-to-end manner, which originally required multiple complex calculations to obtain, thereby facilitating the initialization operation of the monocular SLAM system when the user is in a static state.

[0184] Embodiment 3

[0185] Based on the same inventive concept, Figure 6, a first embodiment of a monocular SLAM initialization device of the present invention is proposed, and the monocular SLAM initialization device can be a virtual device applied to a monocular SLAM initialization device.

[0186] Combine the following Figure 6 The functional module schematic diagram shown in FIG. 1 describes in detail the monocular SLAM initialization device provided in this embodiment, and the device may include:

[0187] A data acquisition module, used to acquire a first frame image and a first feature point of the first frame image;

[0188] An image processing module, used for inputting the first frame image into a pre-trained generative network model to obtain a first homography matrix; wherein the generative network model is constructed based on a convolutional neural network and a generative adversarial network;

[0189] A perspective transformation module, used for performing perspective transformation on the first frame image according to the first homography matrix to obtain a virtual frame image;

[0190] A feature conversion module, used to convert the first feature point to the virtual frame image using the first homography matrix to obtain a second feature point of the virtual frame image;

[0191] An initialization module is used to obtain a first map point using a triangulation algorithm according to the second feature point and the corresponding first feature point.

[0192] Furthermore, the data acquisition module may also be used to acquire a second frame image and a third feature point of the second frame image;

[0193] Correspondingly, the device may further include:

[0194] A feature matching module, used to perform feature point matching on the first frame image and the second frame image with the first frame image as a reference frame, to obtain a first feature point that is successfully matched;

[0195] A map point projection module, used to project the first map point corresponding to the successfully matched first feature point onto the second frame image using the first homography matrix to obtain a projection point in the second frame image;

[0196] A map point screening module is used to determine the first map point corresponding to the projection point that meets a preset condition as a final map point, wherein the preset condition is that the third feature point exists within a preset range centered on the projection point.

[0197] Furthermore, the device may also include:

[0198] A camera pose module is used to obtain the camera pose of the second frame image by minimizing the reprojection error method according to the correspondence between the final map point and the third feature point.

[0199] Furthermore, the device may also include:

[0200] A training data collection module, used to obtain a training data set;

[0201] Network building module, used to build hybrid neural networks;

[0202] Wherein, the hybrid neural network comprises:

[0203] The feature extraction layer is used to extract features from the input image and output feature vectors;

[0204] A convolutional network layer, used for obtaining a generated image of the input image by upsampling according to the feature vector, the convolutional network layer comprising a convolution module, an activation module and a custom module;

[0205] A generating network layer, used for obtaining a homography matrix between the input image and the generated image and a corresponding score through regression prediction according to the feature vector and the generated image;

[0206] A model training module, used to train the hybrid neural network according to the training data set to obtain the generated network model;

[0207] Furthermore, the training data collection module may include:

[0208] An initial data unit, used to obtain a plurality of key frame images, fourth feature points of the plurality of key frame images, and second map points corresponding to the plurality of key frame images;

[0209] A weight calculation unit, used for calculating the observation weight of the second map point according to the multiple key frame images;

[0210] A data determination unit, configured to determine, for each key frame image, a common view frame image of the key frame image and a fifth feature point of the common view frame image;

[0211] A matrix calculation unit, used for obtaining a basic matrix between the key frame image and the common view frame image according to the camera pose of the key frame image and the camera pose of the common view frame image;

[0212] a score calculation unit, configured to obtain a score of the key frame image according to the observation weight, the basic matrix, and a matching relationship between the fourth feature point and the fifth feature point of the key frame image;

[0213] A data set acquisition unit is used to obtain a training data set based on the multiple key frame images, the common view frame images of the multiple key frame images, the second homography matrix of the multiple key frame images and the scores of the multiple key frame images, wherein the second homography matrix is ​​converted from the basic matrix.

[0214] Furthermore, the score calculation unit can be used to:

[0215] Calculating the epipolar line of the fifth feature point according to the basic matrix and the fourth feature point of the key frame image;

[0216] Calculating a matching score between a fourth feature point and the fifth feature point of the key frame image according to a distance between the fifth feature point and the epipolar line;

[0217] The score of the key frame image is obtained according to the observation weight and the matching score.

[0218] Furthermore, the data set acquisition unit can be used to:

[0219] Filtering the multiple key frame images and the common view frame images of the multiple key frame images according to the scores of the multiple key frame images to obtain filtered key frame images and common view frame images;

[0220] sorting the screened common view frame images according to the scores of the screened key frame images to obtain sorted common view frame images;

[0221] Calculating the angle value of the sorted common view frame image according to the rotation matrix between the filtered key frame image and its corresponding filtered common view frame image;

[0222] According to the angle value, inserting the sorted common view frame image into a rotation histogram to obtain a count value of the rotation histogram;

[0223] According to the count value, a final key frame image and its corresponding common view frame image, a second homography matrix and a score are determined to obtain a training data set.

[0224] It should be noted that the functions that can be realized by each module in the monocular SLAM initialization device provided in this embodiment and the corresponding technical effects achieved can refer to the description of the specific implementation methods in each embodiment of the monocular SLAM initialization method of the present invention. For the sake of brevity of the specification, they will not be repeated here.

[0225] Embodiment 4

[0226] Based on the same inventive concept, Figure 2Schematic diagram of the hardware structure, this embodiment provides a monocular SLAM initialization device, the device may include a processor and a memory, the memory stores a monocular SLAM initialization program, when the monocular SLAM initialization program is executed by the processor, all or part of the steps of each embodiment of the monocular SLAM initialization method of the present invention are implemented.

[0227] The monocular SLAM initialization device refers to a terminal device, network device or system device that can achieve network connection. For example, it can be a terminal device such as a mobile phone, computer, tablet computer, embedded industrial computer, or a network device such as a server or cloud platform. It can also be a system device that includes terminal devices such as computers and mobile phones and network devices such as edge servers and cloud servers.

[0228] It can be understood that the monocular SLAM initialization device can also include a communication bus, a user interface and a network interface. Among them, the communication bus is used to realize the connection and communication between these components; the user interface is used to connect to the client and communicate data with the client, and the user interface can include an output unit, such as a display screen, and an input unit, such as a keyboard; the network interface is used to connect to the backend server and communicate data with the backend server, and the network interface can include an input / output interface, such as a standard wired interface and a wireless interface.

[0229] The memory is used to store various types of data, which may include, for example, instructions of any application or method in the monocular SLAM initialization device, and data related to the application. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk, and optionally, the memory may also be a storage device independent of the processor.

[0230] The processor is used to call the monocular SLAM initialization program stored in the memory and execute the monocular SLAM initialization method as described above. The processor can be an application specific integrated circuit (Application Specific Integrated Circuit, referred to as ASIC), a digital signal processor (Digital Signal Processor, referred to as DSP), a digital signal processing device (Digital Signal Processing Device, referred to as DSPD), a programmable logic device (Programmable Logic Device, referred to as PLD), a field programmable gate array (Field Programmable Gate Array, referred to as FPGA), a controller, a microcontroller, a microprocessor or other electronic components, which are used to execute all or part of the steps of each embodiment of the monocular SLAM initialization method as described above.

[0231] Embodiment 5

[0232] Based on the same inventive concept, this embodiment provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a disk, an optical disk, a server, etc., and a computer program is stored on the storage medium, and the computer program can be executed by one or more processors. When the computer program is executed by the processor, all or part of the steps of each embodiment of the monocular SLAM initialization method of the present invention can be implemented.

[0233] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above embodiments are only optional embodiments of the present invention and do not limit the patent scope of the present invention. All equivalent structures or equivalent process changes made by using the contents of the present specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A monocular SLAM initialization method, characterized in that: The method comprises: Acquire a first frame image and a first feature point of the first frame image; Inputting the first frame image into a pre-trained generative network model to perform feature extraction, upsampling and regression prediction through the generative network model to obtain a first homography matrix; wherein the generative network model is constructed based on a convolutional neural network and a generative adversarial network; According to the first homography matrix, performing perspective transformation on the first frame image to obtain a virtual frame image; Using the first homography matrix, converting the first feature point to the virtual frame image to obtain a second feature point of the virtual frame image; A first map point is obtained by using a triangulation algorithm according to the second feature point and the corresponding first feature point.

2. Monocular SLAM initialization method as claimed in claim 1, characterized in that, After the step of obtaining the first map point by using a triangulation algorithm according to the second feature point and the corresponding first feature point, the method further includes: Acquire a second frame image and a third feature point of the second frame image; Taking the first frame image as a reference frame, performing feature point matching on the first frame image and the second frame image to obtain first feature points that are successfully matched; Using the first homography matrix, projecting the first map point corresponding to the successfully matched first feature point onto the second frame image to obtain a projection point in the second frame image; The first map point corresponding to the projection point that meets a preset condition is determined as a final map point, wherein the preset condition is that the third feature point exists within a preset range centered on the projection point.

3. monocular SLAM initialization method as claimed in claim 2, characterized in that, After the step of determining the first map point corresponding to the projection point that meets the preset conditions as the final map point, the method further includes: According to the correspondence between the final map point and the third feature point, the camera pose of the second frame image is obtained by minimizing the reprojection error method.

4. monocular SLAM initialization method as claimed in claim 1, characterized in that, Before the step of inputting the first frame image into a pre-trained generative network model to perform feature extraction, upsampling and regression prediction through the generative network model to obtain a first homography matrix, the method further includes: Get the training dataset; Construct hybrid neural networks; According to the training data set, the hybrid neural network is trained to obtain the generated network model; Wherein, the hybrid neural network comprises: The feature extraction layer is used to extract features from the input image and output feature vectors; A convolutional network layer, used for obtaining a generated image of the input image by upsampling according to the feature vector, the convolutional network layer comprising a convolution module, an activation module and a custom module; A generation network layer is used to obtain a homography matrix between the input image and the generated image and a corresponding score through regression prediction according to the feature vector and the generated image.

5. The monocular SLAM initialization method as claimed in claim 4, wherein: The step of obtaining a training data set comprises: Acquire a plurality of key frame images, fourth feature points of the plurality of key frame images, and second map points corresponding to the plurality of key frame images; Calculating the observation weight of the second map point according to the multiple key frame images; For each key frame image, determining a common view frame image of the key frame image and a fifth feature point of the common view frame image; Obtaining a basic matrix between the key frame image and the common view frame image according to the camera pose of the key frame image and the camera pose of the common view frame image; Obtaining a score of the key frame image according to the observation weight, the basic matrix, and a matching relationship between the fourth feature point and the fifth feature point of the key frame image; A training data set is obtained according to the multiple key frame images, the common view frame images of the multiple key frame images, the second homography matrix of the multiple key frame images and the scores of the multiple key frame images, wherein the second homography matrix is ​​converted from the basic matrix.

6. The monocular SLAM initialization method as claimed in claim 5, wherein: The step of obtaining the score of the key frame image according to the observation weight, the basic matrix, and the matching relationship between the fourth feature point and the fifth feature point of the key frame image comprises: Calculating the epipolar line of the fifth feature point according to the basic matrix and the fourth feature point of the key frame image; Calculating a matching score between a fourth feature point and the fifth feature point of the key frame image according to a distance between the fifth feature point and the epipolar line; The score of the key frame image is obtained according to the observation weight and the matching score.

7. The monocular SLAM initialization method as claimed in claim 5, wherein: The step of obtaining a training data set according to the multiple key frame images, the co-viewing frame images of the multiple key frame images, the second homography matrices of the multiple key frame images, and the scores of the multiple key frame images comprises: Filtering the multiple key frame images and the common view frame images of the multiple key frame images according to the scores of the multiple key frame images to obtain filtered key frame images and common view frame images; sorting the screened common view frame images according to the scores of the screened key frame images to obtain sorted common view frame images; Calculating the angle value of the sorted common view frame image according to the rotation matrix between the filtered key frame image and its corresponding filtered common view frame image; According to the angle value, inserting the sorted common view frame image into a rotation histogram to obtain a count value of the rotation histogram; According to the count value, a final key frame image and its corresponding common view frame image, a second homography matrix and a score are determined to obtain a training data set.

8. A monocular SLAM initialization device, characterized in that: The device comprises: A data acquisition module, used to acquire a first frame image and a first feature point of the first frame image; An image processing module, used for inputting the first frame image into a pre-trained generative network model to perform feature extraction, upsampling and regression prediction through the generative network model to obtain a first homography matrix; wherein the generative network model is constructed based on a convolutional neural network and a generative adversarial network; A perspective transformation module, used for performing perspective transformation on the first frame image according to the first homography matrix to obtain a virtual frame image; A feature conversion module, used to convert the first feature point to the virtual frame image using the first homography matrix to obtain a second feature point of the virtual frame image; An initialization module is used to obtain a first map point using a triangulation algorithm according to the second feature point and the corresponding first feature point.

9. A monocular SLAM initialization device, characterized in that: The device includes a processor and a memory, wherein a monocular SLAM initialization program is stored in the memory, and when the monocular SLAM initialization program is executed by the processor, the monocular SLAM initialization method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by one or more processors, the monocular SLAM initialization method according to any one of claims 1 to 7 is implemented.

Images