An image quality-oriented multi-camera SLAM positioning method and system

Abstract

The present invention discloses an image quality-oriented multi-camera SLAM positioning method and system, the method comprising: acquiring multi-camera images, constructing projection mapping of each camera and transformation relationship from camera to virtual body point; performing image quality assessment on the multi-camera images of the current frame, and obtaining the image with the richest texture according to the impact quality assessment result; constructing an image pyramid, extracting multi-scale features of the image with the richest texture, and dividing it into a uniform image grid so that feature points are evenly distributed on the image, performing feature matching with image features in a reference key frame, forming epipolar geometry constraints, initializing the movement of the image with the richest texture according to the epipolar geometry constraints, establishing a reprojection error with a local map to optimize the movement, and outputting the position and posture of the current multi-camera image according to the transformation relationship from camera to virtual body point corresponding to the image with the richest texture. The present invention efficiently processes multi-camera inputs and can efficiently provide accurate and robust visual positioning.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image quality-oriented multi-camera SLAM positioning method, system, terminal and computer-readable storage medium. Background Art

[0002] In the field of robotics, SLAM (Simultaneous Localization And Mapping,

[0003] The technology of simultaneous localization and mapping plays an important role in providing the robot with an estimate of its own state while providing corresponding perception by reconstructing the surrounding environment. This technology provides basic support for the robot's autonomy, thereby realizing functions such as autonomous navigation, autonomous exploration, and autonomous search. Among them, visual SLAM technology uses visual sensors (monocular cameras, depth cameras, fisheye cameras, etc.) as the main input. The visual sensor can collect a large amount of environmental texture information while maintaining a low load. Thanks to these advantages and the development of the field of computer vision, this technology has received widespread attention and is committed to providing accurate and robust positioning for robots.

[0004] According to the type of visual sensor, in conventional camera configurations, visual SLAM can be divided into monocular visual SLAM, binocular visual SLAM, and RGB-D (RGB+Depth Map, depth image) visual SLAM. The visual SLAM algorithms and systems of these configurations have been extensively studied, and many classic frameworks have been born, showing excellent positioning performance. For unconventional camera configurations, multi-camera SLAM and event camera SLAM have also been studied by some researchers, and corresponding positioning solutions have been provided. For multi-camera SLAM algorithms, their sensor configuration can be described as a rigid platform composed of a set of cameras with fixed internal and external parameters. Under this sensor configuration, the wide field of view can capture rich environmental information to solve the shortcomings of being easily affected by local texture loss, local dynamic interference, and local overexposure or underexposure under traditional configurations. These effects will seriously interfere with positioning performance and even cause positioning failure. On the other hand, the SLAM algorithm based on fisheye and panoramic cameras, which can also expand the field of view, can be arbitrarily arranged with the camera field of view, so that it can be more flexibly adjusted according to task requirements.

[0005] However, the existing multi-camera SLAMs are simply improved based on the traditional SLAM framework, lacking targeted design for multi-camera input. Multi-camera SLAM algorithms based on PTAM (Parallel Tracking And Mapping, a keyframe-based monocular visual SLAM algorithm), ORB-SLAM (ORB: Oriented FAST and Rotated, ORB-SLAM is a 3D positioning and map building algorithm based on ORB features), and VINS-Mono (a robust and versatile monocular visual inertial state estimator) have been proposed one after another. By introducing a unified camera model, feature extraction and matching or optical flow tracking are performed on multi-camera images to achieve multi-camera positioning. SVO2.0 (Sparse Visual Odometry, SVO2.0 is a hybrid odometer) improves the initialization strategy, keyframe selection, and map management of multi-camera SLAM, making the algorithm more efficient. However, these algorithms need to process multi-camera inputs at the front end of the system, which seriously affects the efficiency of the algorithm and the burden on the computing device. Therefore, how to efficiently process multi-camera inputs while maintaining the advantages of multi-camera inputs has become the key to multi-camera SLAM algorithms, enabling them to efficiently provide accurate and robust positioning. On the other hand, existing algorithms require a high degree of synchronization of multi-camera data, which further increases the design burden and reduces its scope of application.

[0006] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0007] The main purpose of the present invention is to provide an image quality-oriented multi-camera SLAM positioning method, system, terminal and computer-readable storage medium, aiming to solve the problems that the existing multi-camera SLAM algorithms in the prior art cannot efficiently process multi-camera inputs and cannot efficiently provide accurate and robust positioning, and the existing algorithms require high synchronization of multi-camera data, which further increases the design burden and reduces its application scope.

[0008] To achieve the above object, the present invention provides an image quality-oriented multi-camera SLAM positioning method, the image quality-oriented multi-camera SLAM positioning method comprising the following steps:

[0009] Acquire multi-camera images, construct projection mapping of each camera and transformation relationship from camera to virtual volume point;

[0010] Perform image quality assessment on the multi-camera images of the current frame, and obtain the image with the richest texture based on the impact quality assessment results;

[0011] Constructing an image pyramid, extracting multi-scale features of the image with the richest texture through the image pyramid, and dividing the image with the richest texture into a uniform image grid so that feature points are evenly distributed on the image;

[0012] The image with the richest texture after feature extraction is feature matched with the image features in the reference key frame to form epipolar geometry constraints. The motion of the image with the richest texture is initialized according to the epipolar geometry constraints, and a reprojection error is established with the local map to optimize the motion. The position and posture of the current multi-camera image are output according to the transformation relationship from the camera to the virtual body point corresponding to the image with the richest texture.

[0013] Optionally, the image quality-oriented multi-camera SLAM positioning method, wherein the step of acquiring multi-camera images and constructing the projection mapping of each camera and the transformation relationship from the camera to the virtual body point specifically includes:

[0014] According to the intrinsic parameters and camera model, the mapping relationship between pixels and map points is constructed;

[0015] Virtual body points are introduced, and the transformation relationship from each camera to the virtual body points is constructed according to external parameters, so that the images of multiple cameras can be unified into the same coordinate system.

[0016] Optionally, the image quality-oriented multi-camera SLAM positioning method, wherein the image quality assessment is performed on the multi-camera image of the current frame, and the image with the richest texture is obtained according to the impact quality assessment result, specifically includes:

[0017] Through lightweight image gradient statistics, the gradient of each pixel is calculated, the gradient division range is counted, and a gradient histogram is formed according to the gradient range;

[0018] The brightness effect is eliminated by normalization, the image texture quality is evaluated according to the gradient histogram, and the image with the largest gradient is selected as the image with the richest texture.

[0019] Optionally, the image quality-oriented multi-camera SLAM positioning method, wherein the position and posture of the current multi-camera image is output according to the transformation relationship from the camera corresponding to the image with the richest texture to the virtual body point, further comprises:

[0020] According to the uncertainty of the current pose estimation, it is judged whether the current local map can provide sufficient reference for the current frame. When the uncertainty is greater than the preset threshold, the current frame is judged as a key frame.

[0021] Optionally, the image quality-oriented multi-camera SLAM positioning method, wherein the uncertainty of the current pose estimation is used to determine whether the current local map can provide sufficient reference for the current frame, and when the uncertainty is greater than a preset threshold, the current frame is determined as a key frame, and then further includes:

[0022] Extract multi-scale features from the remaining images of the current frame, provide all image acquisition information for the local map, and weight each image according to the impact quality assessment results;

[0023] For new features in new keyframes, local map points are constructed using triangulation;

[0024] For a set of key frames and corresponding map points in the local map, a reprojection error is constructed and minimized to optimize the pose of the key frames and the position of the map points in the local map;

[0025] When a new key frame is added, the redundant key frames and map points in the local map are removed, and the local map is updated. The updated local map is used for motion estimation.

[0026] Optionally, the image quality-oriented multi-camera SLAM positioning method, wherein the minimizing the reprojection error to optimize the pose of the key frames and the positions of the map points in the local map, specifically includes:

[0027] ;

[0028] in, represents the optimization variable, represents the minimization of the reprojection error, , represents the poses of all keyframes in the local map, Represents all map points in the local map. Represents the external parameter matrix corresponding to each key frame in the local map, For the The key frame number is The weight represented by the image, For the The key frame number is The image of feature pixels, represents the back-projection mapping, For the The key frame number is The external parameter transformation from camera to body point, For the The pose of the keyframe body point, for The coordinates of the matched map point in the world coordinate system;

[0029] Traversal Keyframes Image feature points, construct the reprojection error by accumulation, and then minimize the reprojection error by gradient descent to complete the optimization of the optimization variables.

[0030] Optionally, in the image quality-oriented multi-camera SLAM positioning method, the removal of redundant key frames and map points in the local map is used to clear redundant data in the local map.

[0031] In addition, to achieve the above-mentioned object, the present invention also provides an image quality-oriented multi-camera SLAM positioning system, wherein the image quality-oriented multi-camera SLAM positioning system comprises:

[0032] Multi-camera image input module, used to obtain multi-camera images, build projection mapping of each camera and transformation relationship from camera to virtual body point;

[0033] A rough image quality assessment module is used to assess the image quality of the multi-camera image of the current frame and obtain the image with the richest texture based on the impact quality assessment results;

[0034] A fine feature extraction module is used to construct an image pyramid, extract multi-scale features of the image with the richest texture through the image pyramid, and divide the image with the richest texture into a uniform image grid so that feature points are evenly distributed on the image;

[0035] The motion estimation module is used to match the image with the richest texture after feature extraction with the image features in the reference key frame to form epipolar geometric constraints, initialize the motion of the image with the richest texture according to the epipolar geometric constraints, establish a reprojection error with the local map to optimize the motion, and output the current multi-camera image posture according to the transformation relationship from the camera to the virtual body point corresponding to the image with the richest texture.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an image quality-oriented multi-camera SLAM positioning program stored in the memory and runnable on the processor, and when the image quality-oriented multi-camera SLAM positioning program is executed by the processor, the steps of the image quality-oriented multi-camera SLAM positioning method as described above are implemented.

[0037] In addition, to achieve the above-mentioned objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image quality-oriented multi-camera SLAM positioning program, and when the image quality-oriented multi-camera SLAM positioning program is executed by a processor, the steps of the image quality-oriented multi-camera SLAM positioning method as described above are implemented.

[0038] In the present invention, multi-camera images are acquired, and projection mapping of each camera and transformation relationship from camera to virtual body point are constructed; image quality assessment is performed on the multi-camera images of the current frame, and the image with the richest texture is obtained according to the impact quality assessment result; an image pyramid is constructed, and multi-scale features of the image with the richest texture are extracted through the image pyramid, and the image with the richest texture is divided into a uniform image grid so that feature points are evenly distributed on the image; feature matching is performed on the image with the richest texture after feature extraction and the image features in the reference key frame to form epipolar geometry constraints, and the movement of the image with the richest texture is initialized according to the epipolar geometry constraints, and a reprojection error is established with the local map to optimize the movement, and the position and posture of the current multi-camera image is output according to the transformation relationship from camera to virtual body point corresponding to the image with the richest texture. The present invention efficiently processes multi-camera inputs and can efficiently provide accurate and robust visual positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a preferred embodiment of the image quality-oriented multi-camera SLAM positioning method of the present invention;

[0040] Figure 2 It is a flow chart of an image quality-oriented multi-camera visual SLAM algorithm in a preferred embodiment of the image quality-oriented multi-camera SLAM positioning method of the present invention;

[0041] Figure 3 It is a schematic diagram of constructing a mapping relationship from pixel points to map points and a transformation relationship from camera to virtual body points in a preferred embodiment of the image quality-oriented multi-camera SLAM positioning method of the present invention;

[0042] Figure 4 It is a structural diagram of a preferred embodiment of the image quality-oriented multi-camera SLAM positioning system of the present invention;

[0043] Figure 5 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] The present invention discloses an image quality-oriented multi-camera SLAM positioning method, comprising a front-end tracking module that uses a coarse-to-fine method to perform a rough quality evaluation on multi-camera images to select high-quality images, performs fine multi-scale feature extraction on the images, and performs motion estimation and positioning based on a local map, thereby improving the front-end tracking efficiency of the algorithm; a back-end optimization module that extracts features from a key frame group and weights the images according to the quality evaluation of the front end, and then uses a weighted local BA optimization (BA is a nonlinear optimization method that optimizes and adjusts camera poses and three-dimensional points by minimizing reprojection errors) to enhance the influence of high-quality images while reducing the interference of low-quality images, thereby improving the accuracy and robustness of the system. In addition, the external parameters between cameras are regarded as dynamic variations and added to the local BA as optimization variables, so that the system can operate normally even when the multi-cameras are weakly synchronized.

[0046] The image quality-oriented multi-camera SLAM positioning method described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the image quality-oriented multi-camera SLAM positioning method includes the following steps:

[0047] Front-end tracking module: The main function of this module is to process multi-camera images and output the pose of the current multi-camera images (body point pose). The specific implementation process is as follows:

[0048] Step S10: Acquire multi-camera images, construct projection mapping of each camera and transformation relationship from camera to virtual body point.

[0049] Specifically, the mapping relationship between pixel points and map points is constructed based on the intrinsic parameters and camera model; the virtual body point is introduced, and the transformation relationship between each camera and the virtual body point is constructed based on the extrinsic parameters (the transformation relationship from camera to body is the transformation from the camera optical center point to the body point. The transformation between two spatial points includes offset and rotation, which corresponds to a transformation matrix in mathematical expression) so that multiple camera images can be unified into the same coordinate system, such as Figure 3 As shown in the figure, so as to establish a connection in the subsequent steps. P(x, y, z) is the map point in the world coordinate system, p(u, v) is the pixel point in the camera coordinate system, π() represents the mapping relationship of projecting p to P, and Tc1b and Tc2b are the transformation matrices from the camera to the virtual body point.

[0050] Among them, projection mapping refers to a function that can project two-dimensional pixels in an image into three-dimensional space. The projection mapping of a camera is related to the intrinsic parameters and camera model of the camera. Camera models include pinhole model, fisheye model, etc. Different models model different projection mappings, and the intrinsic parameters correspond to the corresponding parameters in the function. For example, the projection mapping of a pinhole camera is u=fx*X / Z+cx, v=fy*Y / Z+cy, u, v are two-dimensional pixel coordinates, X, Y, Z are three-dimensional space coordinates, this function is projection mapping, and fx, fy, cx, cy are internal parameters.

[0051] Among them, the primary goal of the visual SLAM algorithm is to estimate the movement of the camera in space. For one camera, visual SLAM estimates the position of the camera's optical center, and for multiple cameras, there are multiple optical centers. Therefore, it is necessary to introduce a spatial point to represent the position of the entire multi-camera group. This spatial point is the body point.

[0052] Among them, the transformation matrix is ​​a 4x4 matrix, which represents the rotation and offset in the three-dimensional space. Through this matrix, a point in the three-dimensional space can be transformed to another point. Here it is used to transform the optical center of the camera to the body point. All point coordinates in the coordinate system based on the optical center of the camera can be transformed into point coordinates in the coordinate system based on the body point through this transformation matrix.

[0053] Step S20: perform image quality assessment on the multi-camera image of the current frame, and obtain the image with the richest texture according to the quality assessment result.

[0054] Specifically, the image quality of the multi-camera image of the current frame is evaluated through lightweight image gradient statistics, which is to traverse each pixel point and calculate the gradient for each pixel. The gradient of the pixel is composed of the vertical gradient and the horizontal gradient. The vertical gradient is the pixel below the pixel minus the pixel above the pixel, and the horizontal gradient is the pixel on the right minus the pixel on the left. After calculating all the gradients, the gradient division range is counted, the number of pixels included in each gradient range is counted, the gradient division range is counted, and a gradient histogram is formed according to the gradient range; then the brightness effect is eliminated through normalization, and the image texture quality is evaluated according to the gradient histogram. The image with a larger gradient indicates that the texture may be richer (the purpose is to obtain the image with the largest gradient among multiple images). Therefore, the image corresponding to the largest gradient is filtered as the image with the richest texture.

[0055] Step S30: construct an image pyramid, extract multi-scale features of the image with the richest texture through the image pyramid, and divide the image with the richest texture into uniform image grids so that feature points are evenly distributed on the image.

[0056] Specifically, an image pyramid is constructed for the above-screened images and multi-scale features are extracted. Then, the image is divided into a uniform image grid so that the feature points (multi-scale is a characteristic of the feature. Extracting features on an image means extracting pixels with features on the image, which are called feature points) are evenly distributed in the image, which can be better connected with the local map construction.

[0057] The construction of the image pyramid is based on the scaling and number of layers. Assuming that a three-layer image pyramid needs to be constructed, each layer of the image is compressed 4 times. The first layer is the original image, and the second layer is a pixel synthesized from every 4 pixels, and the image pixel is ¼ of the original image. The third layer is a pixel synthesized from every 4 pixels of the second layer image, and the image pixel is ¼ of the second layer image. This constitutes a three-layer image pyramid. The image grid divides the image into grids, each grid contains a fixed number of pixels, and the specific number of pixels can be determined by custom parameters. The purpose of the image pyramid is to extract multi-scale features, and the image grid is to make the feature points more evenly distributed on the image.

[0058] It should be noted that multi-scale features are fine features. The fineness here is compared with the coarseness of the previous step. The image quality evaluation in the previous step is a relatively rough process for the purpose of efficiency. The feature extraction in this step is a relatively fine process for the purpose of extracting reliable and accurate features.

[0059] Step S40, feature matching is performed between the image with the richest texture after feature extraction and the image features in the reference key frame to form epipolar geometry constraints, the motion of the image with the richest texture is initialized according to the epipolar geometry constraints, a reprojection error is established with the local map to optimize the motion, and the position and posture of the current multi-camera image is output according to the transformation relationship from the camera to the virtual body point corresponding to the image with the richest texture.

[0060] Specifically, the current image is feature matched with the reference keyframe. The reference keyframe is generally set manually. Here, the latest keyframe can be taken from the keyframes maintained in the local map as the reference keyframe. This keyframe mainly contains all the images of multiple cameras, the features in each image, the position and posture of the body points, etc. The image with the largest gradient screened out in the above steps is feature-matched with the image features in the reference keyframe after feature extraction. There are many methods for feature matching. Generally, the brute force matching method can be used to traverse the features of the current image and compare each feature with the features of the reference image. If the comparison results are similar, the match is successful.

[0061] Then, the motion of the current image is initialized according to the epipolar geometry constraint. After the above feature matching steps, the current image and the reference image will form an epipolar geometry constraint. The constraint can be used to construct a linear equation group, and the motion of the current image can be calculated through the equation group. Then, a reprojection error is established with the local map (the local map contains key frames and map points, and the features of the current image are matched with the map points. For each matched map point, the above projection mapping can be used to project it to the pixel plane of the current image. There is usually a certain position error between the projected pixels and the matched pixels. This error is called the reprojection error) to further optimize the motion and improve the accuracy of the pose estimation. Finally, the pose of the body is restored according to the transformation relationship between the image corresponding to the camera and the body, that is, the current frame pose.

[0062] For a single camera, this step can be omitted. For multiple cameras, there are multiple optical centers. The body point is used to represent the current camera's posture. The motion of the current image is initialized and optimized in the above step. This is the motion of the camera corresponding to the current image. Therefore, the motion of the body can be restored through external parameters, and then the body's posture (that is, the body point posture) can be obtained.

[0063] Furthermore, a key frame judgment is performed: according to the uncertainty of the current pose estimation, whether the current local map can provide sufficient reference for the current frame is judged according to the uncertainty (judged by solving the uncertainty as mentioned above; when the uncertainty is too high, it means that the current local map cannot provide sufficient reference to ensure the accuracy of the pose estimation). When the uncertainty is greater than the preset threshold, the current frame is judged as a key frame.

[0064] Because visual SLAM is a state estimation problem, that is, using visual sensor input to estimate the current pose, and because sensor input noise is ubiquitous and unavoidable, pose estimation becomes a probabilistic problem, that is, there is a certain degree of uncertainty, and uncertainty is used to represent the error range of the estimated value. In particular, the uncertainty of pose estimation can be represented by a covariance matrix, through which the uncertainty of the current pose estimate can be calculated in a certain way.

[0065] Backend optimization module: The main function of this module is to maintain the local map to provide a reference for the front end, optimize the camera external parameters, and reduce the impact of weak synchronization of multi-camera input.

[0066] Keyframe feature extraction and image weighting: In order to retain the advantages of multi-camera configuration, for a keyframe, it contains all multi-camera images, and multi-scale features are extracted for the remaining images of the current frame, providing all image acquisition information for the local map, and providing a more comprehensive reference for the above motion estimation. In addition, each image is weighted according to the impact quality assessment results (a keyframe consists of multiple images, and each image is assigned a corresponding weight based on the above image quality evaluation. This weight will be used in the following local BA optimization) to adjust the impact of the image on the local BA (Bundle Adjustment) optimization.

[0067] Since each frame contains multiple camera images, the image with the largest gradient will be selected in the above steps for feature extraction, matching, and motion estimation. If the current frame is a key frame, feature extraction will be performed on the remaining images in this step.

[0068] Constructing local map points: For new features in the new keyframe, local map points are constructed by triangulation (i.e., new map points are constructed by triangulation). Since the feature points in the new keyframe are two-dimensional pixels and lack depth information, new map points cannot be constructed directly through projection relationships. Therefore, for each feature point of the current image in the current frame, other images with a certain spatial offset from this image are required to observe this feature point. The two feature points can converge at a point in space through projection relationships. At this time, the depth of the two-dimensional pixel point is restored, and then a point in three-dimensional space, i.e., a map point, is constructed. This process is called triangulation.

[0069] Local BA optimization with weights and external parameters: For a set of key frames and corresponding map points in the local map, construct the reprojection error and minimize the reprojection error to optimize the pose of the key frames and the position of the map points in the local map. The reprojection error construction of each image in each frame is related to its weight to adjust the impact of the image on BA optimization. In addition, the camera external parameters are also added to the optimization variables to eliminate the impact of weak synchronization of multi-camera inputs. As shown in the following formula:

[0070] ;

[0071] in, represents the optimization variable, represents the minimization of the reprojection error, , represents the poses of all keyframes in the local map, Represents all map points in the local map. Represents the external parameter matrix corresponding to each key frame in the local map, MRepresents the number of cameras. Note that the number of extrinsic parameter matrices is related to the actual number of cameras used. The purpose of this formula is to minimize the reprojection error by optimizing these variables.

[0072] further, For the The key frame number is The weight represented by the image, For the The key frame number is The image of feature pixels, represents the back-projection mapping, For the The key frame number is The external parameter transformation from camera to body point, For the The pose of the keyframe body point, for The coordinates of the map points in the matched world coordinate system; traverse Keyframes Image feature points, construct the reprojection error by accumulation, and then minimize the reprojection error by gradient descent to complete the optimization of the optimization variables.

[0073] Redundant information removal and local map update: In order to maintain the optimization scale and improve the efficiency of the backend optimization module, whenever a new key frame is added, the redundant key frames and map points in the local map are removed (redundant information removal is a step to clear redundant data in the local map, and ultimately the updated local map performs motion estimation on the newly added ordinary frames), and the local map is updated. The updated local map is used for motion estimation.

[0074] The ultimate goal of the present invention is to perform motion estimation, and the final output is the position and posture of the body point, which can also be said to be the positioning result of the body point.

[0075] The present invention is different from the existing multi-camera SLAM algorithm and has the following innovations (benefits):

[0076] (1) A coarse-to-fine strategy is used to process multi-camera images to solve the shortcomings of high computational overhead and low efficiency when front-end tracking processes multiple images at one time. At the same time, in the key frame addition stage, multi-camera images are packaged into a key frame and inserted into the local map to retain the advantages of multiple cameras. In detail, rough image gradient statistics are used to evaluate image quality, and the best quality image is selected for fine feature extraction and matching, and finally used for motion estimation.

[0077] (2) Using the image evaluation of the above innovations, weights are assigned to each image in the key frame. During local BA optimization, the weight of the reprojection error constructed for each image is adjusted, and then the impact on the back-end optimization is adjusted according to the image quality, making the back-end optimization more accurate.

[0078] (3) The camera extrinsic parameters are incorporated into the optimization variables, and the current camera extrinsic parameters are continuously updated in the local BA optimization, so that the multi-camera SLAM algorithm can still maintain stable performance when the camera input is weakly synchronized.

[0079] The purpose of the present invention is to provide an image quality-oriented multi-camera visual SLAM method in view of the defects of the prior art. The front-end tracking module processes the input of multi-camera images from coarse to fine, selects high-quality images and combines them with local maps for motion estimation, and the back-end optimization module extracts key frame features and weights the images, and then performs BA optimization with weights and external parameters. In this way, the defects of the prior art in which the computational overhead is too large or the operating efficiency is low can be solved, while ensuring that the advantages of multi-camera input can be fully utilized to provide accurate and robust visual positioning. In addition, high synchronization of multi-camera input is not required.

[0080] Furthermore, if Figure 4 As shown, based on the above-mentioned image quality-oriented multi-camera SLAM positioning method, the present invention also provides an image quality-oriented multi-camera SLAM positioning system accordingly, wherein the image quality-oriented multi-camera SLAM positioning system includes:

[0081] A multi-camera image input module 51 is used to obtain multi-camera images, construct projection mapping of each camera and transformation relationship from camera to virtual body point;

[0082] A rough image quality assessment module 52 is used to assess the image quality of the multi-camera image of the current frame, and obtain the image with the richest texture according to the impact quality assessment result;

[0083] A fine feature extraction module 53 is used to construct an image pyramid, extract multi-scale features of the image with the richest texture through the image pyramid, and divide the image with the richest texture into a uniform image grid so that feature points are evenly distributed on the image;

[0084] The motion estimation module 54 is used to perform feature matching between the image with the richest texture after feature extraction and the image features in the reference key frame to form epipolar geometric constraints, initialize the motion of the image with the richest texture according to the epipolar geometric constraints, establish a reprojection error with the local map to optimize the motion, and output the current multi-camera image posture according to the transformation relationship from the camera to the virtual body point corresponding to the image with the richest texture.

[0085] Furthermore, if Figure 5 As shown, based on the above-mentioned image quality-oriented multi-camera SLAM positioning method and system, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 5 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0086] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an image quality-oriented multi-camera SLAM positioning program 40 is stored on the memory 20, and the image quality-oriented multi-camera SLAM positioning program 40 can be executed by the processor 10, thereby realizing the image quality-oriented multi-camera SLAM positioning method in the present application.

[0087] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the image quality-oriented multi-camera SLAM positioning method.

[0088] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0089] In one embodiment, when the processor 10 executes the image quality-oriented multi-camera SLAM positioning program 40 in the memory 20 , the steps of the image quality-oriented multi-camera SLAM positioning method are implemented.

[0090] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image quality-oriented multi-camera SLAM positioning program, and when the image quality-oriented multi-camera SLAM positioning program is executed by a processor, the steps of the image quality-oriented multi-camera SLAM positioning method as described above are implemented.

[0091] In summary, the present invention provides an image quality-oriented multi-camera SLAM positioning method, system, terminal and storage medium, the method comprising: acquiring multi-camera images, constructing projection mapping of each camera and transformation relationship from camera to virtual body point; performing image quality assessment on the multi-camera images of the current frame, and obtaining the image with the richest texture according to the impact quality assessment result; constructing an image pyramid, extracting multi-scale features of the image with the richest texture through the image pyramid, and dividing the image with the richest texture into a uniform image grid so that feature points are evenly distributed on the image; performing feature matching on the image with the richest texture after feature extraction and the image features in the reference key frame to form epipolar geometry constraints, initializing the movement of the image with the richest texture according to the epipolar geometry constraints, establishing a reprojection error with a local map to optimize the movement, and outputting the position and posture of the current multi-camera image according to the transformation relationship from camera to virtual body point corresponding to the image with the richest texture. The present invention efficiently processes multi-camera inputs and can efficiently provide accurate and robust visual positioning.

[0092] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0093] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0094] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An image quality-oriented multi-camera SLAM positioning method, characterized in that: The image quality-oriented multi-camera SLAM positioning method comprises: Acquire multi-camera images, construct projection mapping of each camera and transformation relationship from camera to virtual volume point; The step of acquiring multi-camera images and constructing projection mapping of each camera and a transformation relationship from the camera to the virtual body point specifically includes: According to the intrinsic parameters and camera model, the mapping relationship between pixels and map points is constructed; Introduce virtual body points, and construct the transformation relationship from each camera to the virtual body points based on external parameters, so that the multi-camera images can be unified into the same coordinate system; Perform image quality assessment on the multi-camera images of the current frame, and obtain the image with the richest texture based on the impact quality assessment results; Constructing an image pyramid, extracting multi-scale features of the image with the richest texture through the image pyramid, and dividing the image with the richest texture into a uniform image grid so that feature points are evenly distributed on the image; The image with the richest texture after feature extraction is feature matched with the image features in the reference key frame to form epipolar geometry constraints. The motion of the image with the richest texture is initialized according to the epipolar geometry constraints, and a reprojection error is established with the local map to optimize the motion. The position and posture of the current multi-camera image are output according to the transformation relationship from the camera to the virtual body point corresponding to the image with the richest texture.

2. The image quality-oriented multi-camera SLAM positioning method according to claim 1, characterized in that: The image quality assessment is performed on the multi-camera image of the current frame, and the image with the richest texture is obtained according to the impact quality assessment result, specifically including: Through lightweight image gradient statistics, the gradient of each pixel is calculated, the gradient division range is counted, and a gradient histogram is formed according to the gradient range; The brightness effect is eliminated by normalization, the image texture quality is evaluated according to the gradient histogram, and the image with the largest gradient is selected as the image with the richest texture.

3. The image quality-oriented multi-camera SLAM positioning method according to claim 1, characterized in that: The method outputs the pose of the current multi-camera image according to the transformation relationship from the camera to the virtual body point corresponding to the image with the richest texture, and then further includes: According to the uncertainty of the current pose estimation, it is judged whether the current local map can provide a reference for the current frame. When the uncertainty is greater than the preset threshold, the current frame is judged as a key frame.

4. The image quality-oriented multi-camera SLAM positioning method according to claim 3, characterized in that: According to the uncertainty of the current pose estimation, whether the current local map can provide a reference for the current frame is judged according to the uncertainty. When the uncertainty is greater than a preset threshold, the current frame is judged as a key frame. Then, the following is further included: Extract multi-scale features from the remaining images of the current frame, provide all image acquisition information for the local map, and weight each image according to the impact quality assessment results; For new features in new keyframes, local map points are constructed using triangulation; For a set of key frames and corresponding map points in the local map, a reprojection error is constructed and minimized to optimize the pose of the key frames and the position of the map points in the local map; When a new key frame is added, the redundant key frames and map points in the local map are removed, and the local map is updated. The updated local map is used for motion estimation.

5. The image quality-oriented multi-camera SLAM positioning method according to claim 4, characterized in that: The minimization of the reprojection error to optimize the pose of the key frames and the positions of the map points in the local map specifically includes: ； in, represents the optimization variable, represents the minimization of the reprojection error, , represents the poses of all keyframes in the local map, Represents all map points in the local map. Represents the external parameter matrix corresponding to each key frame in the local map, For the The key frame number is The weight represented by the image, For the The key frame number is The image of feature pixels, represents the back-projection mapping, For the The key frame number is The external parameter transformation from camera to body point, For the The pose of the keyframe body point, for The coordinates of the matched map point in the world coordinate system; Traversal Keyframes Image feature points, construct the reprojection error by accumulation, and then minimize the reprojection error by gradient descent to complete the optimization of the optimization variables.

6. The image quality-oriented multi-camera SLAM positioning method according to claim 5, characterized in that: The removal of redundant key frames and map points in the local map is used to clear redundant data in the local map.

7. An image quality-oriented multi-camera SLAM positioning system, characterized in that: The image quality-oriented multi-camera SLAM positioning system comprises: Multi-camera image input module, used to obtain multi-camera images, build projection mapping of each camera and transformation relationship from camera to virtual body point; The step of acquiring multi-camera images and constructing projection mapping of each camera and a transformation relationship from the camera to the virtual body point specifically includes: According to the intrinsic parameters and camera model, the mapping relationship between pixels and map points is constructed; Introduce virtual body points, and construct the transformation relationship from each camera to the virtual body points based on external parameters, so that the multi-camera images can be unified into the same coordinate system; A rough image quality assessment module is used to assess the image quality of the multi-camera image of the current frame and obtain the image with the richest texture based on the impact quality assessment results; A fine feature extraction module is used to construct an image pyramid, extract multi-scale features of the image with the richest texture through the image pyramid, and divide the image with the richest texture into a uniform image grid so that feature points are evenly distributed on the image; The motion estimation module is used to match the image with the richest texture after feature extraction with the image features in the reference key frame to form epipolar geometric constraints, initialize the motion of the image with the richest texture according to the epipolar geometric constraints, establish a reprojection error with the local map to optimize the motion, and output the current multi-camera image posture according to the transformation relationship from the camera to the virtual body point corresponding to the image with the richest texture.

8. A terminal, characterized in that: The terminal includes: a memory, a processor, and an image quality-oriented multi-camera SLAM positioning program stored in the memory and executable on the processor. When the image quality-oriented multi-camera SLAM positioning program is executed by the processor, the steps of the image quality-oriented multi-camera SLAM positioning method as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an image quality-oriented multi-camera SLAM positioning program, and when the image quality-oriented multi-camera SLAM positioning program is executed by a processor, the steps of the image quality-oriented multi-camera SLAM positioning method according to any one of claims 1 to 6 are implemented.

Images