Visual mapping method, device, storage medium and computer program product

Abstract

The embodiment of the present application provides a visual mapping method, device, storage medium and computer program product, wherein the visual mapping method includes: obtaining image frames taken by a camera and inertial navigation observation data corresponding to the image frames; determining conventional feature points and additional feature points in the image frames; performing online state estimation on the camera pose and the position of conventional feature points in the actual physical space, and establishing online map data according to the results of the online state estimation; performing offline state estimation on the position of additional feature points in the actual physical space, and establishing offline map data according to the results of the offline state estimation; merging the online map data and the offline map data to obtain complete map data. The amount of computation for online real-time positioning and mapping is reduced, the efficiency of online real-time positioning is ensured, and the accuracy of the complete map data is also ensured to be high.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of visual technology, and in particular to a visual mapping method, device, storage medium, and computer program product. Background Art

[0002] Positioning technology is a key technology commonly required in the fields of augmented reality (AR), autonomous driving, robotics, etc. Among positioning technologies, visual positioning technology is widely used due to its low equipment cost and relatively mature technology.

[0003] In visual positioning technology, compared with satellite positioning, network positioning and other technologies, visual simultaneous localization and mapping (V-SLAM) can get rid of the dependence on satellite signals or network communications, realize the state estimation of the device's own position, posture and the position of other elements in the environment in an unknown environment, and obtain positioning results and environmental maps in real time.

[0004] In actual working scenarios based on V-SLAM technology, image acquisition devices such as cameras are often unable to effectively locate due to abnormal conditions such as occlusion and jitter, and need to use map data for relocation to achieve continuous state estimation and positioning. However, in the process of establishing map data, feature points need to be extracted for mapping. If there are too many feature points, the amount of calculation is large, which consumes too many resources and affects the efficiency of real-time positioning. If there are too few feature points, the accuracy of the map data is affected, which will lead to inaccurate relocation. Summary of the invention

[0005] In view of this, embodiments of the present application provide a visual mapping method, device, storage medium, and computer program product to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of an embodiment of the present application, a visual mapping method is provided, comprising: acquiring an image frame taken by a camera, and inertial navigation observation data corresponding to the image frame; determining conventional feature points and additional feature points in the image frame; performing online state estimation of a camera pose and a position of conventional feature points in an actual physical space based on observation data of the position of the conventional feature points in the image frame, and inertial navigation observation data corresponding to the image frame, and establishing online map data according to the result of the online state estimation; performing offline state estimation of the position of the additional feature points in the actual physical space based on observation data of the position of the additional feature points in the image frame, and state data of the camera pose obtained by the online state estimation, and establishing offline map data according to the result of the offline state estimation; and merging the online map data and the offline map data to obtain complete map data.

[0007] According to a second aspect of an embodiment of the present application, a visual mapping device is provided, comprising: an acquisition module, used to acquire image frames taken by a camera, and inertial navigation observation data corresponding to the image frames; a feature point module, used to acquire image frames taken by a camera, and inertial navigation observation data corresponding to the image frames; an online state estimation module, used to perform online state estimation of a camera posture and a position of conventional feature points in an actual physical space based on observation data of positions of conventional feature points in image frames, and inertial navigation observation data corresponding to image frames, and to establish online map data according to the results of the online state estimation; an offline state estimation module, used to perform offline state estimation of positions of additional feature points in an actual physical space based on observation data of positions of additional feature points in image frames, and state data of camera posture obtained by online state estimation, and to establish offline map data according to the results of offline state estimation; and a map module, used to merge online map data and offline map data to obtain complete map data.

[0008] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the visual mapping method described in the first aspect.

[0009] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the visual mapping method as described in the first aspect is implemented.

[0010] According to the visual mapping method, device, storage medium and computer program product provided by the embodiment of the present application, the image frame taken by the camera and the inertial navigation observation data corresponding to the image frame are obtained; the conventional feature points and the additional feature points are determined in the image frame; based on the observation data of the position of the conventional feature points in the image frame and the inertial navigation observation data corresponding to the image frame, the camera posture and the position of the conventional feature points in the actual physical space are estimated online, and online map data is established according to the result of the online state estimation; based on the observation data of the position of the additional feature points in the image frame and the state data of the camera posture obtained by the online state estimation, the position of the additional feature points in the actual physical space is estimated offline, and offline map data is established according to the result of the offline state estimation; the online map data and the offline map data are merged to obtain the complete map data. Because the establishment of online real-time positioning depends on conventional feature points, and conventional feature points are not all feature points, the amount of calculation is reduced, and the efficiency of online real-time positioning is guaranteed, and the complete map data includes online map data and offline map data, and the offline map data is established in an offline state based on the additional feature points, which does not occupy the computing power of online real-time positioning, and ensures that the accuracy of the complete map data is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0012] Figure 1 is a flow chart of a visual mapping method according to Embodiment 1 of the present application;

[0013] Figure 2 A schematic diagram of a conventional feature point according to the first embodiment of the present application;

[0014] Figure 3 is a flow chart of a visual mapping method according to Embodiment 2 of the present application;

[0015] Figure 4 A schematic diagram of a characteristic point according to the second embodiment of the present application;

[0016] Figure 5A is a flow chart of a relocation method according to Embodiment 3 of the present application;

[0017] Figure 5B for Figure 5A A schematic diagram of a feature point extraction, tracking and mapping process in the illustrated embodiment;

[0018] Figure 5Cfor Figure 5A A schematic diagram of instant positioning and conventional feature point map construction in the illustrated embodiment;

[0019] Figure 5D for Figure 5A A schematic diagram of constructing an additional feature point map in the illustrated embodiment;

[0020] Figure 6 is a structural block diagram of a visual mapping device according to Embodiment 4 of the present application;

[0021] Figure 7 This is a schematic diagram of the structure of an electronic device according to Embodiment 5 of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0023] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0024] Embodiment 1

[0025] Reference Figure 1 , shows a flowchart of a visual mapping method according to Embodiment 1 of the present application.

[0026] The visual mapping method of this embodiment includes the following steps:

[0027] Step S101: Acquire image frames captured by a camera and inertial navigation observation data corresponding to the image frames.

[0028] It should be noted that inertial navigation observation data refers to data obtained based on inertial navigation technology. For example, inertial navigation observation data may include camera posture determined by inertial navigation technology, and inertial navigation observation data may be obtained based on sensors such as inertial measurement units (English: Inertial Measurement Unit, IMU). The inertial navigation observation data corresponding to an image frame refers to the inertial navigation observation data measured within the same time period as the image frame. A time period may be 1 second, 2 seconds or 5 seconds, etc., and the length of a time period may be set in advance. It should also be noted that the number of image frames may be one or more, and correspondingly, the number of inertial navigation observation data is also one or more. Image frames and corresponding inertial navigation observation data may be periodically acquired in chronological order. Of course, this is just an exemplary description.

[0029] Step S102: determining regular feature points and additional feature points in the image frame.

[0030] It should be noted that in this application, conventional feature points are feature points used for online positioning and online state estimation, and additional feature points are feature points used for offline state estimation. If there are multiple image frames, after determining the conventional feature points and additional feature points in one image frame, the corresponding conventional feature points and additional feature points in other image frames can be determined. Figure 2 As shown, Figure 2 This is a schematic diagram of conventional feature points according to the first embodiment of the present application. The first image frame contains conventional feature points a1, b1, c1, and the corresponding object points in the actual physical space (in the map) are A, B, and C, that is, a1, b1, c1 are the image points of A, B, and C, respectively. The conventional feature points in the second image frame are the image point a2 corresponding to point A in the second image frame, the image point b2 corresponding to point B in the second image frame, and the image point c2 corresponding to point C in the second image frame. Of course, this is just an example. There are many ways to determine conventional feature points and additional feature points in an image frame. Here, two specific implementation methods are listed for exemplary explanation:

[0031] Optionally, in the first example, determining conventional feature points and additional feature points in an image frame includes: calculating the response value of a point in the image frame, and determining a point whose response value is greater than a preset response threshold as a feature point; dividing the image frame into at least two grid areas, determining the first Nc% of feature points in the grid area whose response values ​​are sorted from large to small as conventional feature points, and determining the remaining feature points in the grid area as additional feature points, where Nc is a number greater than 0 and less than 100. Exemplarily, Nc can be equal to 50, 60, 40, etc. The larger Nc is, the more conventional feature points there are, and the more accurate the online positioning is, but it will also consume more computing power and reduce efficiency. The smaller Nc is, the more conventional feature points there are, and the online positioning accuracy will be reduced, but it will reduce computing power and improve efficiency. Regardless of the value of Nc, the total number of feature points is not affected, that is, the accuracy of the complete map data finally obtained can be guaranteed. Among them, the feature point can be a relatively prominent point in the image frame, and in some application scenarios, the feature point can be an extreme point. The response value of a feature point can represent the prominence of the feature point in the image frame. The larger the response value, the more prominent the feature point. After the image frame is divided into at least two grid areas, some conventional feature points and additional feature points are selected in each grid area, which can ensure that the conventional feature points are distributed more evenly in the image frame, and the distribution of the additional feature points in the image frame is also more evenly distributed, which can improve the accuracy of online positioning and online state estimation, as well as the accuracy of offline state estimation. Optionally, the response value of a point can represent the similarity between the point and the points in its neighborhood. The smaller the similarity (the larger the difference), the larger the response value, and the larger the similarity (the smaller the difference), the smaller the response value. Exemplarily, the average value of the difference between the pixel values ​​of a point and other points in its neighborhood is taken as the response value of the point. Of course, this is just an exemplary explanation.

[0032] Step S103: Based on the observation data of the positions of the conventional feature points in the image frame and the inertial navigation observation data corresponding to the image frame, the camera posture and the positions of the conventional feature points in the actual physical space are estimated online, and online map data is established according to the results of the online state estimation.

[0033] Online map data refers to map data determined by online state estimation. Optionally, the observation data of the position of the conventional feature point in the image frame may include the coordinates of the conventional feature point in the image frame. Here, a specific example is given to illustrate how to perform online state estimation and establish online map data. Based on the observation data of the position of the conventional feature point in the image frame and the inertial navigation observation data corresponding to the image frame, the camera pose and the position of the conventional feature point in the actual physical space are estimated online, and the online map data is established according to the result of the online state estimation, including:

[0034] Based on the observation data of the position of the conventional feature point in the image frame and the inertial navigation observation data corresponding to the image frame, the estimated data of the camera pose and the estimated data of the position of the conventional feature point in the actual physical space are calculated; the residual data is calculated based on the estimated data of the camera pose and the estimated data of the position of the conventional feature point in the actual physical space, and for the same conventional feature point in at least two image frames, when the sum of the residual data of the conventional feature point is minimized, the estimated data of the position of the conventional feature point in the actual physical space is used as the state data of the conventional feature point; and online map data is established based on the state data of at least one conventional feature point.

[0035] Calculating the residual data and determining the state data of the conventional feature points based on the residual data can optimize the error and improve the accuracy of mapping. It should be noted that the estimated data of the camera pose can be determined based on the inertial navigation observation data; the estimated data of the position of the conventional feature points in the actual physical space can be the estimated coordinates of the conventional feature points in the actual physical space, and the state data of the conventional feature points can be the coordinates of the conventional feature points in the actual physical space that are finally determined. It should be noted that if multiple image frames are included in time order, the residual data of the conventional feature points can represent the error between the observation of the current image frame and the observation of the historical image frame. Of course, this is only an exemplary description. Further optionally, the residual data can include at least one of the residuals formed by the marginalization prior constraints of the time window, the residuals of the inertial navigation observation data between two adjacent frames, and the reprojection residuals of the conventional feature points. When the residual data contains multiple residuals, the optimization effect can be improved, so that the state data of the conventional feature points is more accurate, that is, the positioning of the conventional feature points in the actual physical space is more accurate. In addition, for the camera pose, the estimated data of the camera pose when the sum of the residual data of at least two conventional feature points is the smallest for an image frame can be used as the state data of the camera pose. The camera pose can be optimized using the residual to improve the accuracy of the camera pose.

[0036] Step S104: Based on the observation data of the position of the additional feature points in the image frame and the state data of the camera pose obtained by the online state estimation, the position of the additional feature points in the actual physical space is estimated offline, and offline map data is established according to the result of the offline state estimation.

[0037] Offline map data refers to map data determined by offline state estimation. Optionally, a specific example is given here to illustrate how to perform offline state estimation and establish offline map data. Based on the observation data of the position of the additional feature points in the image frame and the state data of the camera pose obtained by online state estimation, the position of the additional feature points in the actual physical space is estimated offline, and the offline map data is established according to the result of the offline state estimation, including:

[0038] According to the observation data of the position of the additional feature point in the image frame and the estimated data of the camera pose, the estimated data of the position of the additional feature point in the actual physical space is calculated; the residual data is calculated based on the estimated data of the position of the additional feature point in the actual physical space, and for the same additional feature point in at least two image frames, when the sum of the residual data of the additional feature point is minimized, the estimated data of the position of the additional feature point in the actual physical space is used as the state data of the additional feature point; and the offline map data is established based on the state data of at least one additional feature point. It should be noted that if multiple image frames are included and sorted by time, the residual data of the additional feature point can represent the error between the observation of the current image frame and the observation of the historical image frame. Of course, this is only an exemplary explanation. The accuracy of the positioning of the additional feature point in the map can be improved by optimizing the state data of the additional feature point using the residual.

[0039] Step S105: merging the online map data and the offline map data to obtain complete map data.

[0040] In combination with the above steps S103 and S104, optionally, the coordinates of feature points in the actual physical space represented by the state data and the estimated data are used as an example for explanation. The online map data may include the coordinates of the regular feature points in the actual physical space, i.e., the positions of the regular feature points in the actual physical space; the offline map data may include the coordinates of the additional feature points in the actual physical space, i.e., the positions of the additional feature points in the actual physical space. In this way, the online map data and the offline map data are merged to obtain the complete map data, i.e., the map data including the coordinates of the regular feature points and the coordinates of the additional feature points.

[0041] Optionally, after obtaining the complete map data, visual repositioning can be performed based on the complete map data when visual positioning anomalies occur. Exemplarily, when it is determined based on the anomaly detection result that a visual positioning anomaly occurs, feature points are extracted for the image frame to be positioned; candidate image frames are obtained from the complete map data, feature points in the candidate image frames are matched with feature points in the image frame to be positioned, and the initial position of the image frame to be positioned is calculated; based on the complete map data, the feature points are reprojected and matched according to the initial position, and the pose of the image frame to be positioned is solved.

[0042] Combined with steps S101-S105, since the establishment of online real-time positioning relies on conventional feature points, which are not all feature points, the amount of calculation is reduced, ensuring the efficiency of online real-time positioning. The complete map data includes online map data and offline map data. The offline map data is established based on additional feature points in an offline state, which does not occupy the computing power of online real-time positioning and ensures the high accuracy of the complete map data.

[0043] Embodiment 2

[0044] Based on the visual mapping method described in the first embodiment, the second embodiment of the present application provides another visual mapping method, which further illustrates the method described in the first embodiment. Figure 3 As shown, the method comprises the following steps:

[0045] Step S301: Acquire image frames captured by a camera and inertial navigation observation data corresponding to the image frames.

[0046] It should be noted that the image frame and the corresponding inertial navigation observation data can be acquired periodically, for example, one image frame and the corresponding inertial navigation observation data are acquired every second, or one image frame and the corresponding inertial navigation observation data are acquired every 0.1 second. The inertial navigation observation data can be acquired using an IMU, of course, this is just an exemplary description.

[0047] Step S302: determining regular feature points and additional feature points in the image frame.

[0048] Specifically, Figure 4 As shown, Figure 4 This is a schematic diagram of a feature point according to the second embodiment of the present application. The image frame is divided into at least two grid areas, and then the response values ​​of the points in the image frame are calculated, and the points whose response values ​​are greater than a preset response threshold are determined as feature points. At least two feature points can be determined in the image frame, and in each grid area, the feature points whose response values ​​are in the first Nc% in order from large to small are determined as regular feature points, and the remaining feature points are determined as additional feature points.

[0049] After step S302, online positioning and online state estimation are performed based on conventional feature points:

[0050] Step S303: Based on the observation data of the positions of the conventional feature points in the image frame and the inertial navigation observation data corresponding to the image frame, the camera posture and the positions of the conventional feature points in the actual physical space are estimated online.

[0051] The online state estimation of conventional feature points can also be called tracking of conventional feature points. Here, a specific example is given for illustration. The camera pose can be estimated using inertial navigation observation data, that is, estimated data representing the camera pose can be obtained. The position (that is, coordinates) of the conventional feature points in the image frame can be determined through the image frame. According to the camera pose determined by the inertial navigation observation data and the position of the conventional feature points in the image frame, the estimated data of the conventional feature points can be estimated by projection in the actual space based on geometric constraints, that is, the estimated coordinates of the conventional feature points in the actual physical space. For the same conventional feature point, each image frame containing the conventional feature point and the corresponding inertial navigation observation data can be used to obtain estimated data of the conventional feature point. There are errors between the estimated data of these conventional feature points, which can be optimized to select the estimated data that minimizes the error between the estimated data of the conventional feature points. In another specific example, the estimation of the camera pose is explained. The camera pose can be determined using inertial navigation observation data. However, for different conventional feature points in the same image frame, the same camera pose but different estimated data of the camera pose will cause different errors in the estimated data of the feature points. Therefore, the estimated data of the camera pose with the smallest sum of the residual data of different conventional feature points should be selected.

[0052] In combination with the above description, optionally, the state quantity is estimated using the method of least squares optimization in the maximum a posteriori sense. An optimization problem is constructed in a time window of a certain length, and the variables to be optimized are:

[0053] χ=[x0,x1,…,x n ,p0,p1,…,p m ]

[0054] Among them, x k is the position, posture, speed and other states of the image frame in the kth cycle, and n is the number of image frames in the time window. l is the 3D position of the lth abnormal map point, and m is the number of all regular feature points observed in all image frames within the time window. The optimization problem constructed based on the maximum a posteriori is as follows:

[0055]

[0056] Among them, min χ It means taking the minimum residual data χ, the first term ||r p -H p χ|| 2 is the residual formed by the marginalization prior constraint of the time window, and {rp, Hp} represents the marginalization prior information; Between the kth frame and the k+1th frame in the time window The residual formed by the IMU observation data, B represents the data set of all IMU observation data; is the lth regular feature point in the jth image frame The visual reprojection residual formed by the observations in the current time window, C represents the set of conventional feature points observed in the current time window. Among them, the specific implementation of each item can be implemented by those skilled in the art through appropriate equations or functions according to actual needs, and the embodiments of the present application are not limited to this. It should be noted that in the specific implementation equations or functions of each of the above items, the state quantities of the IMU bias, conventional image frames and conventional map points (i.e., state estimation results) directly take the initial values ​​without optimization and updating.

[0057] Step S304: determining the state data of the camera pose and the state data of the conventional feature points according to the result of the online state estimation, and establishing the online map data based on the state data of at least one conventional feature point.

[0058] Combined with Formula 1, when the minimum value of the residual data is met, the estimated data of the camera pose is used as the state data of the camera pose, and the estimated data of the conventional feature points is used as the state data of the conventional feature points.

[0059] After step S302, offline state estimation is performed based on the additional feature points:

[0060] Step S305: Based on the observation data of the position of the additional feature point in the image frame and the state data of the camera pose obtained by the online state estimation, the position of the additional feature point in the actual physical space is estimated offline.

[0061] The offline state estimation of additional feature points can also be called tracking of additional feature points. Here, a specific example is given to illustrate. According to the state data of the camera posture and the observation data of the additional feature points (the coordinates of the additional feature points in the image frame), projection is performed in the actual physical space based on geometric constraints to obtain the estimated data of the additional feature points, that is, the estimated coordinates of the additional feature points in the actual physical space. For the same additional feature point, there may be errors between the estimated data obtained by projecting the coordinates in different image frames. Therefore, it is necessary to minimize the sum of the errors to ensure that the coordinates of the additional feature points in the actual physical space are more accurate.

[0062] In combination with the above description, optionally, in one embodiment, the least square optimization method under the maximum likelihood sense is used to solve. For the qth additional feature point in the system, the optimized variable is the estimated data of the additional feature point (i.e., the coordinates of the additional feature point in the actual physical space) p q The corresponding optimization problem is as follows:

[0063]

[0064] in, is the qth regular feature point in the jth image frame Optionally, when the number of image frames of an additional feature point exceeds a preset number, the state estimation of the additional feature point can be optimized, that is, the offline state estimation is performed using Formula 2. Of course, this is only an exemplary description.

[0065] Step S306: determining state data of additional feature points according to the result of offline state estimation, and establishing offline map data based on the state data of at least one additional feature point.

[0066] It should be noted that steps S303 - S304 and steps S305 - S306 may be executed in any order and may also be executed using different threads.

[0067] After step S304 and step S306, the method further includes:

[0068] Step S307: merging the online map data and the offline map data to obtain complete map data.

[0069] Since the establishment of online real-time positioning in this embodiment relies on conventional feature points, which are not all feature points, the amount of calculation is reduced and the efficiency of online real-time positioning is guaranteed. The complete map data includes online map data and offline map data. The offline map data is established in an offline state based on additional feature points, which does not occupy the computing power of online real-time positioning and ensures that the complete map data has a high accuracy.

[0070] The visual mapping method of this embodiment can be executed by any appropriate electronic device with visual mapping capabilities, including but not limited to: mobile terminals (such as mobile phones, PADs, etc.), PCs, and servers.

[0071] Embodiment 3

[0072] This embodiment is based on the method of the above-mentioned embodiment 1 or 2, and describes a relocation method based on the method. Figure 5A As shown, the method comprises the following steps:

[0073] Step S100: relocation map construction.

[0074] The relocalization map is constructed before camera occlusion / fast motion occurs, including the estimation of the position status of regular feature points and additional feature points. Assuming that the feature point tracking is stable during this process, the camera pose and the position of the regular feature point map points can be estimated online, while the position of the additional feature points can be estimated offline, providing a basis for relocalization in advance.

[0075] In the above process, the process of extracting visual feature points for conventional feature points and additional feature points is as follows: Figure 5B As shown, including:

[0076] Process 1: Feature point extraction and descriptor calculation.

[0077] In this embodiment, visual feature points are divided into two categories: conventional feature points (such as Figure 4 The star symbol) and additional feature points (such as Figure 4 In the same frame image, there is no intersection between the conventional feature point set and the additional feature point set. This embodiment uses the same method to uniformly extract feature points and divide them into conventional feature points and additional feature points, but does not limit the specific types of feature points. In order to improve the accuracy of real-time positioning and mapping, it is necessary to ensure that the two types of feature points are evenly distributed in the image. To this end, this embodiment divides the image frame into multiple grids, and extracts a certain proportion of conventional feature points and additional feature points in each grid.

[0078] In one embodiment, a certain score value threshold is set, and N FAST corner points are extracted from the entire image. According to the size of the score value, the top N points with the largest score value in each grid are selected. c % FAST corner points are used as regular feature points; the remaining N e % FAST corner points are used as additional feature points.

[0079] Process 2: Based on the conventional feature points and the additional feature points, conventional feature point tracking and additional feature point tracking are performed.

[0080] After the conventional feature points and additional feature points are extracted, the descriptors of the feature points are calculated, and this embodiment does not specify the descriptor type. In one embodiment, the ORB descriptor is used as a local feature for description. The conventional feature points and additional feature points are tracked between frames respectively. This embodiment does not specify the tracking method. In one embodiment, the LK optical flow method is used for inter-frame tracking. According to the inter-frame correlation relationship of the conventional feature points, combined with other sensors, the position of the conventional feature points and the real-time posture of the camera can be estimated in real time to obtain the current camera posture and the conventional feature point map; according to the results of camera positioning and the inter-frame tracking of the additional feature points, the position of the additional feature points is estimated in another thread to obtain an additional feature point map.

[0081] Process 3: Real-time positioning and construction of conventional feature point maps (real-time positioning and mapping).

[0082] Real-time positioning and regular feature point map construction solve the problem of estimating the real-time pose state of the camera in the world coordinate system and the position state estimation of regular feature points in the world coordinate system. Real-time positioning and regular feature point map construction can be reduced to a system state estimation problem, such as Figure 5C As shown, the observations of the system may include (but are not limited to) the observations of the camera on regular feature points over a period of time. (Solid line as shown in symbol 5), observation of inertial navigation module (IMU) (As shown by the dotted line in symbol 4), the state quantity may include (but is not limited to) the posture state of the camera (as shown in symbol 1) over a period of time, the position state of the conventional feature point (as shown in symbol 2), and the bias state of the inertial navigation module. It is worth noting that in order to control the consumption of computing power, in the real-time positioning and the construction of the conventional feature point map, the observation quantity does not include the camera's observation of the conventional feature point (as shown in symbol 3), and the state quantity does not include the position state of the conventional feature point.

[0083] This embodiment does not limit the solution method of the state estimation problem. In one embodiment, the state quantity is estimated using the method of least squares optimization under the maximum a posteriori sense. The optimization problem is constructed in a time window of a certain length, and the variables to be optimized are:

[0084] χ=[x0,x1,…,x n ,p0,p1,…,p m ]

[0085] Among them, x k is the position of the camera at time k, and n is the number of camera moments in the time window. l is the 3D position of the lth regular feature point, and m is the total number of regular feature points observed in all regular frames in the time window. As mentioned above, the state quantity may also include camera speed, IMU bias, etc., which are not shown in the above formula. The optimization problem constructed based on the maximum a posteriori is:

[0086]

[0087] The meaning of each parameter in the above formula can be described in reference to Formula 1. For example, the first term is the residual formed by the marginalization prior constraint of the sliding time window. It is the residual of the IMU observation between the kth frame and the k+1th frame in the sliding time window. is the visual reprojection residual formed by observing the lth regular feature point in the jth frame camera.

[0088] Process 4: Additional feature point map construction based on offline mapping.

[0089] Figure 5DThe figure shows a schematic diagram of the construction of the additional feature point map. In process 3, the camera pose as shown in symbol 7 has been estimated. The construction of the additional feature point map solves the problem of estimating the position state of the additional feature points based on the camera pose and the camera's observation of the additional feature points from multiple perspectives. It can be reduced to a state estimation problem. The state quantity that needs to be estimated is the position of the additional feature point (as shown in symbol 6); the observation quantity is the camera's observation of the additional feature point (As shown in symbol 8).

[0090] This embodiment does not limit the solution method for the aforementioned state estimation problem. In one embodiment, the least squares optimization method under the maximum likelihood sense is used for solution. For the qth additional feature point map feature point in the system, the optimized variable is p q The corresponding optimization problem is

[0091]

[0092] The meaning of each parameter in the above formula can be described in the above formula 2, such as, Represents the residual formed by observing the point in the jth frame image.

[0093] It should be noted that, unlike the real-time solution of complex optimization problems in real-time positioning and conventional feature point map construction, the construction of the additional feature point map does not need to be carried out in real time, and the state estimation optimization problem corresponding to different feature points can be solved in different threads. In one embodiment, when the number of image frames in which a certain additional feature point is observed exceeds a certain threshold, the state estimation optimization problem of the feature point is solved.

[0094] Step S200: performing relocation based on the constructed relocation map.

[0095] When the camera fails to track feature points due to occlusion or rapid movement, the relocalization process begins. In this process, the feature points of the relocalization frame are first extracted. According to a certain method (such as Bag of words), the key frames in the map are initially associated to obtain candidate matching key frames; then, according to the feature point descriptors and geometric constraints, the feature points in the candidate key frames are matched with the feature points in the relocalization frame, and the rough position of the current frame is calculated accordingly; finally, all other feature points in the map are reprojected and matched according to the rough position to obtain the final relocalization frame-map point association relationship, and the pose of the relocalization frame is solved. Feature point matching and pose estimation in relocalization can be done using mainstream algorithms such as ORB-SLAM and VINS-MONO, which will not be described in detail here.

[0096] Through this embodiment, effective visual relocation can be performed based on the constructed relocation map.

[0097] Embodiment 4

[0098] In combination with the above-mentioned embodiment 1 and embodiment 4, refer to Figure 6 , shows a structural block diagram of a visual mapping device according to Embodiment 3 of the present application.

[0099] The visual mapping device 50 of this embodiment includes: an acquisition module 501, which is used to acquire image frames taken by a camera and inertial navigation observation data corresponding to the image frames; a feature point module 502, which is used to determine conventional feature points and additional feature points in the image frames; an online state estimation module 503, which is used to perform online state estimation of the camera posture and the position of the conventional feature points in the actual physical space based on the observation data of the position of the conventional feature points in the image frames and the inertial navigation observation data corresponding to the image frames, and establish online map data according to the results of the online state estimation; an offline state estimation module 504, which is used to perform offline state estimation of the position of the additional feature points in the actual physical space based on the observation data of the position of the additional feature points in the image frames and the state data of the camera posture obtained by the online state estimation, and establish offline map data according to the results of the offline state estimation; a map module 505, which is used to merge the online map data and the offline map data to obtain complete map data.

[0100] Optionally, the feature point module 502 is used to calculate the response value of the point in the image frame, and determine the point whose response value is greater than a preset response threshold as a feature point; divide the image frame into at least two grid areas, and determine the first Nc% of the feature points in the grid area sorted from large to small in response value as regular feature points, and determine the remaining feature points in the grid area as additional feature points, where Nc is a number greater than 0 and less than 100.

[0101] Optionally, the online state estimation module 503 is used to calculate the estimated data of the camera pose and the estimated data of the position of the conventional feature points in the actual physical space based on the observation data of the positions of the conventional feature points in the image frames and the inertial navigation observation data corresponding to the image frames; calculate the residual data based on the estimated data of the camera pose and the estimated data of the positions of the conventional feature points in the actual physical space, and for the same conventional feature point in at least two image frames, calculate the estimated data of the position of the conventional feature point in the actual physical space when the sum of the residual data of the conventional feature point is minimized; and establish online map data based on the estimated data of the position of at least one conventional feature point in the actual physical space.

[0102] Optionally, the online state estimation module 503 estimates the camera pose when the sum of residual data of at least two conventional feature points is minimized for an image frame.

[0103] Optionally, the residual data includes at least one of a residual formed by a time window marginalization prior constraint, a residual of inertial navigation observation data between two adjacent frames, and a reprojection residual of a conventional feature point.

[0104] Optionally, the offline state estimation module 504 is used to calculate the estimated data of the position of the additional feature points in the actual physical space based on the observation data of the position of the additional feature points in the image frame and the estimated data of the camera pose; calculate the residual data based on the estimated data of the position of the additional feature points in the actual physical space, and for the same additional feature point in at least two image frames, calculate the estimated data of the position of the additional feature point in the actual physical space when the sum of the residual data of the additional feature point is minimized; and establish offline map data based on the estimated data of the position of at least one additional feature point in the actual physical space.

[0105] Optionally, the visual mapping device 50 also includes a repositioning module 506, which is used to extract feature points of the image frame to be positioned when it is determined that a visual positioning anomaly occurs based on the anomaly detection result; obtain a candidate image frame from the complete map data, match the feature points in the candidate image frame with the feature points in the image frame to be positioned, and calculate the initial position of the image frame to be positioned; based on the complete map data, reproject and match the feature points according to the initial position, and solve the posture of the image frame to be positioned.

[0106] The visual mapping device of this embodiment is used to implement the corresponding visual mapping methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, the functional implementation of each module in the visual mapping device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be repeated here.

[0107] Embodiment 5

[0108] In combination with the above-mentioned embodiment 1 and embodiment 3, refer to Figure 7 , shows a schematic diagram of the structure of an electronic device according to Embodiment 5 of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0109] like Figure 7 As shown, the electronic device may include: a processor (processor) 602 , a communication interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0110] in:

[0111] The processor 602 , the communication interface 604 , and the memory 606 communicate with each other via a communication bus 608 .

[0112] The communication interface 604 is used to communicate with other electronic devices or servers.

[0113] The processor 602 is used to execute the program 610, and specifically can execute the relevant step S in the above-mentioned visual mapping method embodiment.

[0114] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0115] The processor 602 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0116] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0117] Program 610 can be specifically used to enable processor 602 to perform the following operations: obtain image frames taken by a camera, and inertial navigation observation data corresponding to the image frames; determine conventional feature points and additional feature points in the image frames; based on the observation data of the positions of conventional feature points in the image frames, and the inertial navigation observation data corresponding to the image frames, perform online state estimation of the camera posture and the positions of conventional feature points in the actual physical space, and establish online map data according to the results of the online state estimation; based on the observation data of the positions of additional feature points in the image frames, and the state data of the camera posture obtained by the online state estimation, perform offline state estimation of the positions of additional feature points in the actual physical space, and establish offline map data according to the results of the offline state estimation; merge the online map data and the offline map data to obtain complete map data.

[0118] In an optional embodiment, program 410 is also used to enable processor 402 to calculate the response value of the point in the image frame, and determine the point whose response value is greater than a preset response threshold as a feature point; divide the image frame into at least two grid areas, and determine the first Nc% of feature points in the grid area sorted from large to small in response value as regular feature points, and determine the remaining feature points in the grid area as additional feature points, where Nc is a number greater than 0 and less than 100.

[0119] In an optional embodiment, the program 410 is also used to enable the processor 402 to calculate the estimated data of the camera pose and the estimated data of the position of the conventional feature points in the actual physical space based on the observation data of the position of the conventional feature points in the image frame and the inertial navigation observation data corresponding to the image frame; calculate the residual data based on the estimated data of the camera pose and the estimated data of the position of the conventional feature points in the actual physical space, and for the same conventional feature point in at least two image frames, the estimated data of the position of the conventional feature point in the actual physical space when the sum of the residual data of the conventional feature point is minimized; and establish online map data based on the estimated data of the position of at least one conventional feature point in the actual physical space.

[0120] In an optional implementation, the program 410 is further configured to enable the processor 402 to estimate the camera pose when the sum of residual data of at least two conventional feature points is minimized for an image frame.

[0121] In an optional implementation, the residual data includes at least one of a residual formed by a time window marginalization prior constraint, a residual of inertial navigation observation data between two adjacent frames, and a reprojection residual of a conventional feature point.

[0122] In an optional embodiment, the program 410 is also used to enable the processor 402 to calculate the estimated data of the position of the additional feature points in the actual physical space based on the observation data of the position of the additional feature points in the image frame and the estimated data of the camera pose; calculate the residual data based on the estimated data of the position of the additional feature points in the actual physical space, and for the same additional feature point in at least two image frames, calculate the estimated data of the position of the additional feature point in the actual physical space when the sum of the residual data of the additional feature point is minimized; and establish offline map data based on the estimated data of the position of at least one additional feature point in the actual physical space.

[0123] In an optional embodiment, the program 410 is also used to enable the processor 402 to extract feature points of the image frame to be located when it determines that a visual positioning anomaly has occurred based on the anomaly detection result; obtain a candidate image frame from the complete map data, match the feature points in the candidate image frame with the feature points in the image frame to be located, and calculate the initial position of the image frame to be located; based on the complete map data, reproject and match the feature points according to the initial position, and solve the posture of the image frame to be located.

[0124] The specific implementation of each step in program 410 can refer to the corresponding description of the corresponding steps in the above-mentioned visual mapping method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0125] Through the electronic device of this embodiment, because the online real-time positioning relies on conventional feature points, which are not all feature points, the amount of calculation is reduced and the efficiency of the online real-time positioning is guaranteed. The complete map data includes online map data and offline map data. The offline map data is established in an offline state based on additional feature points, which does not occupy the computing power of the online real-time positioning and ensures that the complete map data has a high accuracy.

[0126] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0127] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the visual mapping method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the visual mapping method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the visual mapping method shown herein.

[0128] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0129] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A visual mapping method, comprising: Acquire image frames captured by a camera and inertial navigation observation data corresponding to the image frames; Determining conventional feature points and additional feature points in the image frame, wherein the conventional feature points are feature points used for online positioning and online state estimation, and the additional feature points are feature points used for offline state estimation; Based on the observation data of the position of the conventional feature point in the image frame and the inertial navigation observation data corresponding to the image frame, the camera posture and the position of the conventional feature point in the actual physical space are estimated online, and online map data is established according to the result of the online state estimation; Based on the observation data of the position of the additional feature point in the image frame and the state data of the camera pose obtained by online state estimation, an offline state estimation is performed on the position of the additional feature point in the actual physical space, and offline map data is established according to the result of the offline state estimation; The online map data and the offline map data are merged to obtain complete map data.

2. The method according to claim 1, wherein: The determining of conventional feature points and additional feature points in the image frame comprises: Calculating the response value of the point in the image frame, and determining the point whose response value is greater than a preset response threshold as a feature point; The image frame is divided into at least two grid areas, and the first Nc% of feature points in the grid area whose response values ​​are sorted from large to small are determined as the regular feature points, and the remaining feature points in the grid area are determined as the additional feature points, where Nc is a number greater than 0 and less than 100.

3. The method according to claim 1, wherein: The method of performing online state estimation on the camera pose and the position of the conventional feature points in the actual physical space based on the observation data of the position of the conventional feature points in the image frame and the inertial navigation observation data corresponding to the image frame, and establishing online map data according to the result of the online state estimation includes: Calculate the estimated data of the camera pose and the estimated data of the position of the conventional feature point in the actual physical space according to the observation data of the position of the conventional feature point in the image frame and the inertial navigation observation data corresponding to the image frame; Residual data is calculated based on the estimated data of the camera pose and the estimated data of the position of the conventional feature point in the actual physical space, and for the same conventional feature point in at least two of the image frames, when the sum of the residual data of the conventional feature point is minimized, the estimated data of the position of the conventional feature point in the actual physical space is used as the state data of the conventional feature point; The online map data is established based on the status data of at least one of the regular feature points.

4. The method according to claim 3, wherein: The method further comprises: For one of the image frames, when the sum of the residual data of at least two of the conventional feature points is minimized, the estimated data of the camera pose is used as the state data of the camera pose.

5. The method according to claim 3, wherein: The residual data includes at least one of the residual formed by the time window marginalization prior constraint, the residual of the inertial navigation observation data between two adjacent frames, and the reprojection residual of the conventional feature point.

6. The method according to claim 1, wherein: The method of performing offline state estimation on the position of the additional feature point in the actual physical space based on the observation data of the position of the additional feature point in the image frame and the estimated data of the camera pose obtained by online state estimation, and establishing offline map data according to the result of offline state estimation includes: Calculating estimated data of the position of the additional feature point in the actual physical space according to the observed data of the position of the additional feature point in the image frame and the estimated data of the camera pose; Residual data is calculated based on the estimated data of the position of the additional feature point in the actual physical space, and for the same additional feature point in at least two of the image frames, when the sum of the residual data of the additional feature point is minimized, the estimated data of the position of the additional feature point in the actual physical space is used as the state data of the additional feature point; The offline map data is established based on the status data of at least one of the additional feature points.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: When it is determined according to the abnormality detection result that a visual positioning abnormality occurs, feature points are extracted from the image frame to be positioned; Acquire a candidate image frame from the complete map data, match feature points in the candidate image frame with feature points in the image frame to be located, and calculate an initial position of the image frame to be located; Based on the complete map data and according to the initial position, the feature points are reprojected and matched, and the position and posture of the image frame to be positioned are solved.

8. A visual mapping device, comprising: An acquisition module is used to acquire image frames taken by a camera and inertial navigation observation data corresponding to the image frames; A feature point module, used to determine conventional feature points and additional feature points in the image frame, wherein the conventional feature points are feature points used for online positioning and online state estimation, and the additional feature points are feature points used for offline state estimation; An online state estimation module, for performing online state estimation on the camera pose and the position of the conventional feature points in the actual physical space based on the observation data of the position of the conventional feature points in the image frame and the inertial navigation observation data corresponding to the image frame, and establishing online map data according to the result of the online state estimation; an offline state estimation module, configured to perform offline state estimation on the position of the additional feature point in the actual physical space based on the observation data of the position of the additional feature point in the image frame and the state data of the camera pose obtained by online state estimation, and to establish offline map data according to the result of offline state estimation; A map module is used to merge the online map data and the offline map data to obtain complete map data.

9. A storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, wherein when the computer program product is executed by a processor, the computer program product implements the method according to any one of claims 1 to 7.

Images