A monocular vision calibration method and computer readable storage medium

Abstract

The present invention relates to the field of navigation technology, and provides a monocular vision calibration method and a computer-readable storage medium, wherein an original image is collected by a monocular camera, semantic information in the original image is extracted, and output posture information and output three-dimensional scene information are obtained by fitting based on the semantic information, the original image, and a plurality of images to be fitted in a time sequence, and the posture and coordinates of the monocular camera are calibrated based on the relative relationship between the output posture information and the output three-dimensional scene information. Through the monocular vision calibration method of the present invention, the traditional monocular vision SLAM technology and the text recognition technology are combined together, which can not only improve the accuracy of the monocular camera posture estimation, but also generate a semantic map that is convenient for users to use.

Description

Technical Field

[0001] The present invention relates to the field of navigation technology, and in particular to a monocular vision calibration method and a computer-readable storage medium. Background Art

[0002] Visual SLAM (Simultaneous Localization and Mapping) is a technology that estimates self-motion and models scenes at the same time. It has been widely used in the fields of self-driving cars, augmented reality, virtual reality, robot navigation, etc. Visual SLAM attempts to solve the problem of how to determine the trajectory of an intelligent agent's motion through the pictures taken by the camera when it moves in an unknown environment, and at the same time build a map of the surrounding environment. Traditional visual SLAM uses very little semantic information in positioning and mapping, so it is limited in many application scenarios. Combining traditional SLAM with semantic information can improve the practicality and robustness of the system, and is more in line with human cognition of exploring unknown environments.

[0003] Obtaining a more accurate feature matching relationship is a crucial component of monocular SLAM. Traditional monocular SLAM only relies on the matching relationship between a limited number of feature points, which may not necessarily obtain accurate camera pose estimation, resulting in low accuracy of the generated map. Moreover, the generated map is a sparse point cloud map, which has low practicality from the perspective of user interaction. Therefore, it is urgent to provide a monocular vision calibration method and a computer-readable storage medium to at least solve the above problems. Summary of the invention

[0004] The object of the present invention is to provide a monocular vision calibration method and a computer-readable storage medium, which can at least partially overcome the deficiencies in the prior art.

[0005] According to one aspect of the present invention, a monocular vision calibration method is provided, comprising the following steps:

[0006] Capturing an original image through a monocular camera, and extracting semantic information from the original image;

[0007] Based on the semantic information and the original image, obtaining initial pose information of the monocular camera and initial coordinate information of the original image;

[0008] Continuously collecting a plurality of images to be fitted by the monocular camera in time sequence, and obtaining the pose information to be fitted of the monocular camera and the coordinate information to be fitted of the image to be fitted based on the uniform velocity model, the initial pose information and the initial coordinate information;

[0009] Determine the number of the to-be-fitted images containing the semantic information, and when the number of the to-be-fitted images containing the semantic information is not less than six, fit the initial pose information and the to-be-fitted pose information to obtain output pose information, and fit the initial coordinate information and the to-be-fitted coordinate information to obtain output three-dimensional scene information;

[0010] The posture and coordinates of the monocular camera are calibrated based on the relative relationship between the output posture information and the output three-dimensional scene information.

[0011] Preferably, before acquiring the original image by the monocular camera, the method further includes: calibrating the intrinsic parameter matrix and distortion parameters of the monocular camera.

[0012] Preferably, the semantic information is a set of pixels having text characteristics in the original image.

[0013] Preferably, fitting the initial coordinate information and the coordinate information to be fitted to obtain output three-dimensional scene information includes: fitting based on the initial coordinate information and the coordinate information to be fitted to obtain multiple semantic planes; fitting based on the semantic planes to obtain the three-dimensional scene information.

[0014] Preferably, fitting based on the initial coordinate information and the coordinate information to be fitted to obtain a plurality of semantic planes comprises: considering the initial coordinate information containing the semantic information and the coordinate information to be fitted to be on the same plane to obtain the semantic planes.

[0015] Preferably, calibrating the posture and coordinates of the monocular camera based on the relative relationship between the output posture information and the output three-dimensional scene information includes: obtaining a reprojection error factor based on the output three-dimensional scene information; and calibrating the coordinates of the monocular camera through the reprojection error factor.

[0016] Preferably, a distance factor is obtained based on the output three-dimensional scene information, and the position and posture of the monocular camera is calibrated based on the distance factor and the reprojection error factor.

[0017] Preferably, based on the distance factor and the reprojection error factor, the pose of the monocular camera is calibrated by a factor graph optimization algorithm.

[0018] Preferably, the factor graph optimization algorithm is constructed based on the G2O library.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the monocular vision calibration method as described in any one of the above items is implemented.

[0020] The present invention provides a monocular vision calibration method, which collects original images through a monocular camera, extracts semantic information from the original images, and obtains output posture information and output three-dimensional scene information based on the semantic information, the original images, and a plurality of images to be fitted in a time sequence, and calibrates the posture and coordinates of the monocular camera based on the relative relationship between the output posture information and the output three-dimensional scene information. Through the monocular vision calibration method of the present invention, the traditional monocular vision SLAM technology and the text recognition technology are combined together, which can not only improve the accuracy of the monocular camera posture estimation, but also generate a semantic map that is convenient for users to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0022] Figure 1 It is a flow chart of the monocular vision calibration method according to the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. For ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0024] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0025] The monocular camera in the embodiment of the present application mainly refers to a monocular digital camera, which can be a pinhole camera or a wide-angle camera and other settings. The reason why this application is aimed at monocular cameras is that a considerable part of the existing equipment, such as vehicle-mounted cameras or cameras carried on drones, are monocular cameras. If they are converted into binocular cameras in order to improve positioning accuracy, the cost of the conversion is relatively high. One of the purposes of this application is to rely on the old monocular camera to make software modifications to the positioning performance of the entire drone or vehicle, which can achieve a significant improvement in positioning accuracy and provide a semantic map that can be directly applied to the drone or vehicle. In addition, this method can also be applied to occasions such as smart glasses with a monocular camera.

[0026] like Figure 1 As shown, the present invention provides a monocular vision calibration method, comprising the following steps:

[0027] S101: collecting an original image through a monocular camera and extracting semantic information from the original image;

[0028] S102: Based on the semantic information and the original image, initial pose information of the monocular camera and initial coordinate information of the original image are obtained;

[0029] S103: Continuously collecting a plurality of images to be fitted in time sequence through a monocular camera, and obtaining the pose information to be fitted of the monocular camera and the coordinate information to be fitted of the image to be fitted based on the uniform velocity model, the initial pose information and the initial coordinate information;

[0030] S104: determining the number of images to be fitted that contain semantic information, and when the number of images to be fitted that contain semantic information is not less than six, fitting the initial pose information and the pose information to be fitted to obtain output pose information, and fitting the initial coordinate information and the coordinate information to be fitted to obtain output three-dimensional scene information;

[0031] S105: Calibrate the position and coordinates of the monocular camera based on the relative relationship between the output position information and the output three-dimensional scene information.

[0032] In process S101, the original image can be a color image or a black and white image, but its clarity should be sufficient to distinguish semantic information, especially which pixels belong to semantic information and which pixels belong to which part of the semantic information. Therefore, the original image in this application has certain requirements for clarity. Extracting semantic information

[0033] In processing S101, software such as Mask TextSpotter can be used to complete the task of detecting and extracting the two-dimensional text information in the picture taken by the camera at the current moment. There are many software that can complete semantic recognition, which will not be described here one by one, and those skilled in the art should be familiar with them. After obtaining whether each pixel point in the current picture belongs to a text feature, and which text feature each pixel point corresponds to, as a preferred implementation method, the two-dimensional text information set (L) in a single picture can be defined as:

[0034]

[0035] Among them l t The t-th two-dimensional text information represented by the word w contained in the text information t And the vertex pixel coordinates p of the polygonal bounding box of the text information in the image t T represents the total number of two-dimensional text information detected in the current picture. When T=-1, it means that no text information is detected in the current picture.

[0036] In processing S102, the initial coordinate information of the original image can be retrieved from an existing map based on the semantic information identified in processing S101. This can be a general navigation map or a special map for a smaller scene, such as a three-dimensional navigation map inside an office building. By retrieving the semantic information, a rough initial coordinate information can be quickly obtained, such as which point on the map the original image corresponds to. However, due to the inaccuracy of the positioning of the monocular camera, there is still a certain gap to confirm the precise position of the monocular camera. The initial pose information can also be determined at the same time. For example, the pixel points belonging to the same semantic information are regarded as being in the same plane, and the semantic information image formed after fitting is compared with the pre-stored semantic information image, so as to obtain a rough initial pose of the current monocular camera. These are all well known to those skilled in the art and will not be described in detail.

[0037] In processing S103, the uniform speed model refers to a model that continues to move at a uniform speed in the direction of the initial movement of the monocular camera based on the initial coordinate information and the initial posture, according to an empirically set parameter or the average speed at the previous moment. This is because the frequency of acquisition and processing of the monocular camera is actually very high. In processing S104, when 6 images to be fitted containing the above-mentioned semantic information are acquired, an output posture information can be output. This process may be very short and does not require very precise confirmation of the speed changes in the middle. The use of a uniform speed model can effectively reduce the amount of calculation.

[0038] In processing S104, in the previous processing, the initial pose and initial coordinate information of the camera have been initialized using the matching relationship of feature points with text semantic labels, and the pose information to be fitted of the monocular camera at each moment thereafter is obtained through the uniform velocity model and new coordinate information to be fitted is generated, and it is continuously detected whether the number of images to be fitted belonging to the semantic information exceeds six. When the number is detected to be more than six, in a preferred implementation method, a random consistency sampling algorithm can be used to fit a plane to the spatial point set. Here, as a preferred implementation method, a semantic plane can be fitted first, and then the semantic plane can be fitted to obtain the output three-dimensional scene information.

[0039] The specific implementation method may be that the geometric expression of the semantic plane obtained by fitting is:

[0040] π=(n x ,n y ,n z ,) T (2.2)

[0041] Among them, π represents the semantic plane obtained by fitting, (n x ,n y ,nz ) T It represents the unit normal vector corresponding to the plane, and d represents the distance from the origin to the plane. After fitting the semantic plane, the above semantic information is used again to construct the spatial plane into output three-dimensional scene information. The output three-dimensional scene information is defined as:

[0042] Π k = {W k ,π k ,Q k ,Y k} (2.3)

[0043] Among them, k represents the kth semantic plane, W k It represents the words contained in the semantic information corresponding to the semantic plane, π k It is the geometric expression of the space plane corresponding to the kth semantic plane, Q k It represents the coordinates of the center point corresponding to the kth semantic plane, Y k It represents the set of map points corresponding to the kth semantic plane.

[0044] In this way, an output three-dimensional scene information tightly coupled with semantic information is obtained, which can provide multiple composite functions such as semantic information guidance while assisting positioning.

[0045] In processing S104, fitting the initial pose information and the pose information to be fitted to obtain the output pose information can be based on a factor graph optimization algorithm. The semantic information in the use scene of the present invention can provide supplementary clues for the camera pose estimation. On the one hand, the semantic information contains a large amount of feature point information, which can help estimate the camera pose. On the other hand, the semantic information can be regarded as a plane with a specific label attribute. By treating the spatial map points belonging to the same text area as being on the same plane, it can be further optimized to obtain better camera pose estimation information. In view of the above-mentioned different influences, the present invention tightly couples the plane semantic information to the factor graph optimization model, and calculates a more accurate camera pose estimation by jointly optimizing the reprojection error factor and the point-to-plane distance error factor. The above process is well known to those skilled in the art and will not be elaborated here.

[0046] The present invention provides a monocular vision calibration method, which collects original images through a monocular camera, extracts semantic information from the original images, and obtains output posture information and output three-dimensional scene information based on the semantic information, the original images, and a plurality of images to be fitted in a time sequence, and calibrates the posture and coordinates of the monocular camera based on the relative relationship between the output posture information and the output three-dimensional scene information. Through the monocular vision calibration method of the present invention, the traditional monocular vision SLAM technology and the text recognition technology are combined together, which can not only improve the accuracy of the monocular camera posture estimation, but also generate a semantic map that is convenient for users to use.

[0047] As a preferred implementation, before processing S101, the intrinsic parameter matrix and distortion parameters of the monocular camera are calibrated. The specific implementation of the calibration process is as follows:

[0048] 1) Perform intrinsic calibration on the camera to obtain the camera’s distortion parameters and intrinsic matrix

[0049]

[0050] Where [x, y] is the coordinates of the normalized plane point, [x distorted ,y distorted ] is the distorted coordinate, k1, k2, k3, p1, p2 are distortion terms, and r is the distance between any point on the plane and the origin of the coordinate system;

[0051]

[0052] P is the camera internal parameter matrix, where f is the camera focal length, [O x ,O y ] is the main optical axis point.

[0053] This can help the subsequent calibration process after completing the calibration of the intrinsic parameter matrix and distortion parameters.

[0054] As a preferred implementation, in processing S101, the semantic information is a set of pixels having text characteristics in the original image.

[0055] As a preferred implementation, in processing S104, fitting the initial coordinate information and the coordinate information to be fitted to obtain output three-dimensional scene information includes: fitting based on the initial coordinate information and the coordinate information to be fitted to obtain multiple semantic planes; fitting based on the semantic planes to obtain three-dimensional scene information. Among them, fitting based on the initial coordinate information and the coordinate information to be fitted to obtain multiple semantic planes includes: considering the initial coordinate information containing semantic information and the coordinate information to be fitted to be on the same plane to obtain semantic planes.

[0056] As a preferred implementation method, in processing S105, calibrating the posture and coordinates of the monocular camera based on the relative relationship between the output posture information and the output three-dimensional scene information includes: obtaining a reprojection error factor based on the output three-dimensional scene information; and calibrating the coordinates of the monocular camera through the reprojection error factor.

[0057] The specific implementation process can be: The reprojection error factor is defined as:

[0058] e i,j =p i,j -Proj(T c P i,j ) (3.1)

[0059] where e i,j represents the reprojection error factor, P i,j It represents the coordinate information of the jth spatial point at time i, p i,j It means P i,j The pixel coordinates of the feature points in the corresponding image, the Proj function represents the projection function of the camera, T c It represents the estimated pose of the camera at the current moment.

[0060] As a preferred implementation method, based on outputting three-dimensional scene information, a distance factor is obtained, and based on the distance factor and a reprojection error factor, the position and posture of the monocular camera is calibrated.

[0061] The specific implementation method can be: the distance factor from the point to the plane (e k,m ) is defined as:

[0062] e k,m =Π k P k,m (3.2)

[0063] Where P k,m represents the homogeneous representation of the spatial coordinates of the mth map point belonging to the kth semantic plane, Π k The geometric expression corresponding to the k-th three-dimensional semantic plane represented by .

[0064] As a preferred implementation method, the position and posture of the monocular camera are calibrated based on the distance factor and the reprojection error factor by a factor graph optimization algorithm, wherein the factor graph optimization algorithm can be constructed based on the G2O library.

[0065] The specific implementation method can be:

[0066] The final camera pose information is obtained by jointly optimizing the reprojection error factor and the distance factor from the point to the plane. The loss function (C) of the least squares model in the factor graph is defined as:

[0067]

[0068] Among them, λ1 represents the weight value of the reprojection error factor, λ2 represents the weight value of the point-to-plane distance factor, and ρ h represents the robust Huber function, e i,j represents the reprojection error factor, e k,m It represents the factor of the distance from the space point to the plane, Ω i,j represents the covariance matrix of the reprojection error factor, Ω k,m It represents the covariance matrix of the point-to-plane distance factor.

[0069] The optimization problem for solving the camera pose parameters is defined as:

[0070]

[0071] in It represents the estimated pose information of the camera that needs to be solved in the end, T c It is the initial value of the camera pose estimation in the factor graph optimization. The present invention uses the G2O library to construct this factor graph optimization problem and adopts the Levenberg-Marquardt (LM) algorithm to solve it, and finally obtains a low-error camera pose estimation.

[0072] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0073] The present application also provides a computer-readable medium having a computer program stored thereon, which implements the monocular visual calibration method described above when executed by a processor. The computer-readable medium includes permanent and non-permanent, removable and non-removable media, and the media can implement information storage by any method or technology. The information can be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carriers.

[0074] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Claims

1. A monocular vision calibration method, characterized in that , including the following steps: Capturing an original image through a monocular camera, and extracting semantic information from the original image; Based on the semantic information and the original image, initial pose information of the monocular camera and initial coordinate information of the original image are obtained; wherein the initial coordinate information is represented as retrieval information of the semantic information in a preset map; The monocular camera continuously collects a plurality of images to be fitted in time sequence, and obtains the pose information to be fitted of the monocular camera and the coordinate information to be fitted of the image to be fitted based on the uniform speed model, the initial pose information and the initial coordinate information; wherein the uniform speed model is a model that moves at a uniform speed toward the initial movement direction of the monocular camera according to set parameters or historical average speed based on the initial coordinate information and the initial pose information; Determine the number of the to-be-fitted images containing the semantic information, and when the number of the to-be-fitted images containing the semantic information is not less than six, fit the initial pose information and the to-be-fitted pose information to obtain output pose information, and fit multiple semantic planes based on the initial coordinate information and the to-be-fitted coordinate information; and fit the semantic planes to obtain three-dimensional scene information; Based on the three-dimensional scene information, a reprojection error factor is obtained; and the coordinates of the monocular camera are calibrated by the reprojection error factor; A distance factor is obtained based on the three-dimensional scene information, and the position and posture of the monocular camera is calibrated based on the distance factor and the reprojection error factor.

2. The monocular vision calibration method according to claim 1, characterized in that: Before the original image is collected by the monocular camera, the method further includes: calibrating the intrinsic parameter matrix and distortion parameters of the monocular camera.

3. The monocular vision calibration method according to claim 1, characterized in that: The semantic information is a set of pixels in the original image that have text characteristics.

4. The monocular vision calibration method according to claim 1, characterized in that: The fitting based on the initial coordinate information and the coordinate information to be fitted to obtain a plurality of semantic planes includes: considering the initial coordinate information containing the semantic information and the coordinate information to be fitted to be on the same plane to obtain the semantic planes.

5. The monocular vision calibration method according to claim 1, characterized in that: Based on the distance factor and the reprojection error factor, the position and posture of the monocular camera are calibrated by a factor graph optimization algorithm.

6. The monocular vision calibration method according to claim 5, characterized in that: The factor graph optimization algorithm is constructed based on the G2O library.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the monocular vision calibration method according to any one of claims 1 to 6 is implemented.

Images