Visual-inertial semi-dense reconstruction method and device, electronic equipment and storage medium

Abstract

The application provides a visual inertial semi-dense reconstruction method, which comprises the following steps: constructing a quadtree of a square for a submarine exploration image, selecting a minimum square from the quadtree of the square, identifying a center pixel point of the minimum square, and constructing a polar segment of the center pixel point; using a preset pyramid optical flow method to identify a first matching pixel point matched with the center pixel point, identifying a second matching pixel point matched with the center pixel point from the polar segment, selecting a third matching pixel point matched with the center pixel point from a square in which the polar segment is located, and obtaining a third matching point pair between the center pixel point and the third matching pixel point; performing depth analysis on the first matching point pair, the second matching point pair and the third matching point pair to obtain an analysis depth, and splicing each analysis depth in the analysis depth to obtain a scene semi-dense reconstruction result of the submarine exploration image. The application can realize a semi-dense scene reconstruction method suitable for a deep-sea robot and optimized for an old semi-dense scene reconstruction algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of marine oil and gas exploration and development, and particularly relates to a visual-inertial semi-dense reconstruction method and device, an electronic device and a storage medium. BACKGROUND

[0002] Marine oil and gas exploration and development has great development prospects. Due to the complex and changeable environment of deep-sea diving exploration operations, there are many uncertain factors, so marine exploration and development faces higher risks and difficulties. Deep-sea robots are extreme operation robots applied to deep water environments. The large demand for underwater operations makes the deep-sea robots not only obtain more accurate trajectory positioning, but also hope to have more accurate and robust three-dimensional reconstruction methods to help them recover the three-dimensional information of the underwater scene and provide effective prior information for underwater operations.

[0003] Historically, RGB-D cameras can help SLAM systems with positioning and mapping. The main reason for dense mapping is that RGB-D cameras can provide depth information for each pixel in the image frame, and the depth information can help recover the scale and positioning. The RGB information obtained by the RGB-D camera can help the RGB-D SLAM system obtain more color information. In three-dimensional reconstruction, the obtained 3D point cloud is more suitable for human visual cognition.

[0004] Although RGB-D SLAM has many advantages in three-dimensional reconstruction, the huge pixel data obtained by the RGB-D camera becomes a burden. Taking a 1080p image as an example, the RGB information attached to each pixel plus the depth information, the data volume of one image frame can reach the order of ten million, which brings great challenges to the long-term maintenance and management of the system.

[0005] Visual SLAM systems can be classified into direct methods and feature point methods according to the image information used. There are hundreds of thousands of pixels in each image, but only a few hundred feature points are extracted each time. The feature point method mainly relies on key point extraction and descriptor calculation in the front end, and ignores other useful image information except feature points. Therefore, the feature point method can only achieve sparse reconstruction under normal circumstances and cannot meet the three-dimensional reconstruction needs of the surrounding environment. With the emergence of some open source projects of direct method SLAM, they can overcome the defects of large data management of RGB-D SLAM, and the reconstruction density can also meet part of the real needs, so they gradually become an important part of visual odometry calculation method and become mainstream.

[0006] The direct method overcomes the problem of low image information utilization rate of the feature point method, mainly calculates the camera motion according to the image pixel information, and can be divided into sparse, semi-dense and dense according to the number of pixels used, which can help to recover the relatively dense structure and is more conducive to the application of scene interaction and the like.

[0007] The dense visual SLAM method aims to estimate the depth of each pixel, including high gradient and low gradient, but this also causes a large amount of data, which burdens the system. Unlike dense visual SLAM, semi-dense visual SLAM only uses high gradient image pixels, because these pixels are the only pixels that produce reliable matches.

[0008] LSD-SLAM proposes a direct monocular SLAM algorithm, which maintains and tracks a global map containing a key frame pose graph and a related probabilistic semi-dense depth map, realizes CPU-level semi-dense reconstruction, considers the uncertainty of triangulation, and adopts a novel filtering form for depth estimation. The LSD-SLAM algorithm adopts the direct method for tracking, and therefore has the advantages of being insensitive to feature point loss, but also covers the shortcomings of the direct method: sensitive to camera exposure, fast motion easily leading to loss, in addition, in the loop detection part, LSD-SLAM still needs to rely on the feature point method, and has not completely got rid of the calculation of feature points.

[0009] DSO is a high-precision direct visual odometry that does not include loop detection function, in order to deal with the problem that the direct method is easily affected by light, it adopts photometric calibration, considers exposure time, lens vignetting and nonlinear response function, and calculates the position of camera pose and map points by minimizing photometric error, and there are many improved works of DSO later, LDSO retains the characteristics of DSO that it can use any image pixel with sufficient gradient, and adds the function modules of loop detection and pose graph optimization, Stereo DSO is a binocular direct sparse odometry, which is used for semi-dense reconstruction of large-scale environment from binocular camera, and has the functions of loop closure and global BA.

[0010] DSM is a completely direct monocular VSLAM method, which allows direct use of photometric error formula to reuse existing map information, DSM adopts a coarse-to-fine optimization scheme, which can increase the convergence radius of PBA. This strategy allows the use of rich geometric information provided by observable points. In addition, DSM uses a local map covisibility window to select active key frames that observe the same scene area (even if they are not close in time).

[0011] Visual-inertial SLAM also has the function requirement of three-dimensional reconstruction, but there are not many existing open source algorithms: VI-DSO is improved based on DSO, adopts a novel initialization strategy, adds scale and gravity direction in the model, and performs joint optimization after initialization, jointly estimates camera pose and semi-dense scene geometry by minimizing photometric error and IMU measurement error, but it is not open source, the existing VI-Stereo-DSO has open source code, but it adds a wrong binocular residual, causing memory leak, ORB-SLAM3 is an open source system that can use monocular, binocular and RGB-D cameras to perform visual, visual-inertial and multi-map SLAM, one of its innovations is the multi-map system, which relies on a new location recognition method that can improve recall rate, so that ORB-SLAM3 can survive in long-term poor visual information: When it loses, it will start a new map, and when it re-visits the previous map, the map will be seamlessly merged with the previous map, but its disadvantage is that the reconstructed map is too sparse.

[0012] The camera has the advantages of small volume, low cost and rich information acquisition, and has become a research hotspot in visual SLAM, but visual SLAM is easily affected by motion state and photometric change, resulting in unsatisfactory effect, and it is also difficult to apply in underwater environment, in addition, monocular visual SLAM cannot restore scale information, and there are few existing researches on underwater environment application of visual-inertial SLAM, the large demand of underwater operation makes the visual-inertial SLAM system obtain more accurate trajectory positioning, and at the same time, more accurate and robust three-dimensional reconstruction method is needed to help it recover the three-dimensional information of underwater scene, and provide effective prior information for underwater operation, in order to ensure the understanding of the scene, the lowest requirement of three-dimensional reconstruction is semi-dense reconstruction, but the visual-inertial SLAM system often discards the old semi-dense scene to obtain more accurate device positioning information, so it cannot meet the semi-dense reconstruction requirement of the scene. Figure Three

[0013] Therefore, there is an urgent need for a semi-dense scene reconstruction method for deep-sea robots which optimizes the old semi-dense scene reconstruction algorithm. SUMMARY

[0014] In order to solve the above problems, the present application provides a visual-inertial semi-dense reconstruction method, device, electronic equipment and storage medium, which can realize a semi-dense scene reconstruction method for deep-sea robots which optimizes the old semi-dense scene reconstruction algorithm.

[0015] In the first aspect, the present application provides a visual-inertial semi-dense reconstruction method, comprising the following steps:

[0016] ​Collecting a subsea exploration image in a vision front-end of a visual-inertial SLAM system, calculating a pixel difference between a pixel square and a neighborhood pixel square of the pixel square in the subsea exploration image, constructing a quad-tree of the pixel square of the subsea exploration image based on the pixel difference, selecting a minimum pixel square from the quad-tree of the pixel square;

[0017] Identifying a center pixel point of the minimum pixel square, and constructing a epipolar segment of the center pixel point;

[0018] When a number of pixel points contained in the epipolar segment is less than a first preset number, identifying a first matching pixel point matched with the center pixel point by using a preset pyramid optical flow method, and obtaining a first matching point pair between the center pixel point and the first matching pixel point;

[0019] When the number of pixel points contained in the epipolar segment is not less than the first preset number and less than a second preset number, identifying a second matching pixel point matched with the center pixel point from the epipolar segment, and obtaining a second matching point pair between the center pixel point and the second matching pixel point;

[0020] When the number of pixel points contained in the epipolar segment is not less than the second preset number, selecting a third matching pixel point matched with the center pixel point from a block where the epipolar segment is located, and obtaining a third matching point pair between the center pixel point and the third matching pixel point;

[0021] Based on pixel point motion information of the subsea exploration image in the visual-inertial SLAM system, performing depth analysis on the first matching point pair, the second matching point pair and the third matching point pair to obtain an analysis depth of the center pixel point, and splicing each analysis depth in the analysis depth to obtain a scene semi-dense reconstruction result of the subsea exploration image.

[0022] In one embodiment, the calculating the pixel difference between the pixel square and the neighborhood pixel square of the pixel square in the subsea exploration image comprises:

[0023] Dividing a pixel square in the subsea exploration image;

[0024] Calculating the pixel difference between the pixel square and the neighborhood pixel square of the pixel square by using the following formula:

[0025]

[0026] Wherein, β represents the pixel difference, x represents a pixel value of a center pixel point in the pixel square, represents a pixel mean value of adjacent pixel points in four directions of the center pixel point in the pixel square.

[0027] In one embodiment, the constructing the quadtree of the square of the ocean exploration image based on the pixel distance comprises:

[0028] When the pixel distance is greater than a preset threshold, taking a center pixel point in a pixel square corresponding to the pixel distance as a center, the ocean exploration image is divided into four grids, and the four grids are taken as second-level quadtree nodes;

[0029] When the size of the four grids is not less than a preset size, the four grids are subjected to multi-level grid division to obtain multi-level grids, and the multi-level grids are taken as multi-level quadtree nodes;

[0030] When the size of the multi-level grids is less than the preset size, the multi-level grids are taken as quadtree leaf nodes;

[0031] Based on the center pixel point, the second-level quadtree nodes, the multi-level quadtree nodes and the quadtree leaf nodes, the quadtree of the square is determined.

[0032] In one embodiment, the constructing the epipolar segment of the center pixel point comprises:

[0033] Obtaining a first visual point and a second visual point of the center pixel point;

[0034] Taking a position with a minimum distance from the center pixel point as a starting point;

[0035] From the starting point, a depth range segment of the center pixel point is constructed in a direction from the first visual point to the position with the minimum distance;

[0036] Identifying a visual interval of the second visual point to the depth range segment;

[0037] In the visual interval, a straight line segment parallel to the depth range segment is queried, and the straight line segment is taken as the epipolar segment.

[0038] In one embodiment, the identifying a second matching pixel point matching the center pixel point from the epipolar segment comprises:

[0039] Collecting pixel points contained in the epipolar segment;

[0040] Calculating a similarity degree between the center pixel point and the pixel points contained in the epipolar segment;

[0041] When the similarity degree is greater than a preset similarity degree, a pixel point corresponding to the similarity degree and contained in the epipolar segment is taken as the second matching pixel point.

[0042] In one embodiment, the third matching pixel point matching the center pixel point is selected from a block in which the epipolar line segment is located, comprising:

[0043] A first block is selected from a neighborhood of the center pixel point, and a second block with the same size as the first block is extracted from a block in which the epipolar line segment is located;

[0044] A difference value between the first block and the second block is calculated using the following formula:

[0045] S(A, B) SSD =∑ i,j (A(i, j)-B(i, j)) 2

[0046] wherein S(A, B) SSD represents the difference value between the first block and the second block, A(i, j) represents a gray value of the first block after removing the average value of each small block, (i, j) represents coordinates in the block, and B(i, j) represents a gray value of the second block after removing the average value of each small block;

[0047] When the difference value is less than a preset difference value, a center point pixel in the second block corresponding to the difference value is taken as the third matching pixel point.

[0048] In one embodiment, based on the pixel point motion information of the subsea exploration image in the visual-inertial SLAM system, a depth analysis is performed on the first matching point pair, the second matching point pair, and the third matching point pair to obtain an analysis depth of the center pixel point, comprising:

[0049] A depth interval of the center pixel point is calculated using the following formula:

[0050]

[0051]

[0052] wherein (s1, s2) represents the depth interval, s1 represents a depth interval corresponding to a center pixel point in a certain matching point pair of the first matching point pair, the second matching point pair, and the third matching point pair, s2 represents a depth interval corresponding to a matching pixel point in a certain matching point pair of the first matching point pair, the second matching point pair, and the third matching point pair, a matrix composed of R and t represents a transformation matrix from a world coordinate system to a camera coordinate system, x1 represents coordinates of the center pixel point in a certain matching point pair of the first matching point pair, the second matching point pair, and the third matching point pair, and x2 represents coordinates of the matching pixel point in a certain matching point pair of the first matching point pair, the second matching point pair, and the third matching point pair.

[0053] Based on the depth interval, coordinates of central pixel points in the first matching point pair, the second matching point pair and the third matching point pair are converted to obtain analysis depth of the central pixel points.

[0054] In a second aspect, the present application provides a visual-inertial semi-dense reconstruction device, characterized in that the device comprises:

[0055] A square selection module is configured to collect a submarine exploration image in a visual front end of a visual-inertial SLAM system, calculate pixel differences between a pixel square and neighbor pixel squares of the pixel square in the submarine exploration image, construct a square quad tree of the submarine exploration image based on the pixel differences, and select a minimum square from the square quad tree.

[0056] A epipolar line construction module is configured to identify a central pixel point of the minimum square and construct an epipolar line segment of the central pixel point.

[0057] A first identification module is configured to, when a number of pixel points contained in the epipolar line segment is less than a first preset number, identify a first matching pixel point matched with the central pixel point by using a preset pyramid optical flow method to obtain a first matching point pair between the central pixel point and the first matching pixel point.

[0058] A second identification module is configured to, when the number of pixel points contained in the epipolar line segment is not less than the first preset number and less than a second preset number, identify a second matching pixel point matched with the central pixel point from the epipolar line segment to obtain a second matching point pair between the central pixel point and the second matching pixel point.

[0059] A third selection module is configured to, when the number of pixel points contained in the epipolar line segment is not less than the second preset number, select a third matching pixel point matched with the central pixel point from a block where the epipolar line segment is located to obtain a third matching point pair between the central pixel point and the third matching pixel point.

[0060] A depth splicing module is configured to, based on pixel point motion information of the submarine exploration image in the visual-inertial SLAM system, perform depth analysis on the first matching point pair, the second matching point pair and the third matching point pair to obtain analysis depth of the central pixel points, splice each analysis depth in the analysis depth to obtain a scene semi-dense reconstruction result of the submarine exploration image.

[0061] In a third aspect, the present application provides an electronic device, characterized in that the electronic device comprises:

[0062] at least one processor; and

[0063] a memory in communication with the at least one processor; wherein

[0064] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the visual-inertial semi-dense reconstruction method as described.

[0065] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the visual-inertial semi-dense reconstruction method as described.

[0066] Compared with the prior art, the embodiments of the present application have at least the following advantages:

[0067] The embodiments of the present application first calculate the pixel difference between the pixel squares in the submarine exploration image and the neighborhood pixel squares of the pixel squares, for dividing the image frame into blocks using the quadtree strategy, so that the pixels in the same block have similar pixel values, and the pixel values in the leaf nodes, that is, the pixels in the smallest blocks, are obviously different from other blocks adjacent thereto and have high image gradients, which can meet the requirements of three-dimensional reconstruction. Secondly, the embodiments of the present application construct the epipolar segment of the center pixel point, for querying the matching points of the center pixel point by using the epipolar search method. Further, the embodiments of the present application perform depth analysis on the first matching point pair, the second matching point pair and the third matching point pair based on the motion information of the pixel points in the submarine exploration image in the visual-inertial SLAM system, for estimating the spatial positions of the feature points by using camera motion in order to achieve good semi-dense reconstruction effect on the VI-SLAM system. With the assistance of the front-end IM (inlier-outlier matching) algorithm, the visual-inertial SLAM has obtained relatively accurate motion information, so that the motion information can be used to perform depth estimation on the matching point pairs obtained by the epipolar search and the block matching strategy. For the pixels obtained in the monocular system, the depth information of the pixels cannot be solved by simply relying on one image frame, so it is necessary to estimate the depth information by using the triangulation method under different viewing angles. Therefore, the visual-inertial semi-dense reconstruction method, device, electronic equipment and storage medium provided by the embodiments of the present application can realize the semi-dense scene reconstruction method suitable for deep-sea robots, which optimizes the old semi-dense scene reconstruction algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0068] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, the other drawings can be obtained based on these drawings without any creative effort.

[0070] Figure 1 A flowchart of a visual-inertial semi-dense reconstruction method provided by an embodiment of the present application is shown in FIG. 1.

[0071] Figure 2 A functional diagram of a visual-inertial SLAM system of a visual-inertial semi-dense reconstruction method provided by an embodiment of the present application is shown in FIG. 2. Figure 1

[0072] Figure 3 A diagram of a quadtree of a visual-inertial semi-dense reconstruction method provided by an embodiment of the present application is shown in FIG. 3. Figure 1

[0073] Figure 4 A diagram of a polar line segment of a visual-inertial semi-dense reconstruction method provided by an embodiment of the present application is shown in FIG. 4. Figure 1

[0074] A diagram of a depth interval of a visual-inertial semi-dense reconstruction method provided by an embodiment of the present application is shown in FIG. 5. Figure 5 Figure 1 A module diagram of a visual-inertial semi-dense reconstruction device provided by an embodiment of the present application is shown in FIG. 6.

[0075] Figure 6 An internal structure diagram of an electronic device for implementing a visual-inertial semi-dense reconstruction method provided by an embodiment of the present application is shown in FIG. 7.

[0076] Figure 7 DETAILED DESCRIPTION

[0077] It should be understood that the specific implementation described herein is only used to explain the present application, and is not used to limit the present application.

[0078] ​​​​This invention provides a visual-inertial semi-dense reconstruction method. The executing entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this invention: a server, a terminal, etc. In other words, the visual-inertial semi-dense reconstruction method can be executed by software or hardware installed on a terminal device or a server device. The software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0079] Example 1

[0080] See Figure 1 The diagram shown is a flowchart illustrating a visual-inertial semi-dense reconstruction method according to an embodiment of the present invention. Figure 1 The visual-inertial semi-dense reconstruction methods described in the text include:

[0081] S1. Acquire underwater exploration images from the visual front end of the visual inertial SLAM system, calculate the pixel difference between the pixel squares in the underwater exploration images and the neighboring pixel squares of the pixel squares, construct a quadtree of the underwater exploration images based on the pixel difference, and select the smallest square from the quadtree.

[0082] In this embodiment of the invention, the visual inertial SLAM system refers to a system that uses a camera to complete the perception of the environment. SLAM stands for Simultaneous Localization and Mapping. The functions of the visual inertial SLAM system include acquiring images, extracting features from the images, constructing a map based on the extracted features, updating the map constructed in the previous step based on the subsequent motion trajectory, determining the current position based on the constructed map and the robot's motion trajectory, and querying matching buildings in the constructed map based on the buildings passed by the current path.

[0083] See Figure 2 As shown, this is one embodiment of the present invention. Figure 1 A functional schematic diagram of a visual-inertial SLAM system based on a visual-inertial semi-dense reconstruction method is provided. Figure 2 In this context, IMU stands for Inertial Measurement Unit, which is mainly used to detect and measure data such as acceleration and rotational motion.

[0084] Further, the embodiment of the present application calculates pixel difference between a pixel square in the submarine exploration image and a neighborhood pixel square of the pixel square, so as to divide the image frame into blocks using a quadtree strategy, so that pixels in the same block have similar pixel values, and pixel values in the leaf node, that is, the smallest block, are obviously different from other blocks adjacent thereto, have high image gradient, and can meet the requirement of three-dimensional reconstruction.

[0085] In an embodiment of the present application, the calculation of the pixel difference between the pixel square in the submarine exploration image and the neighborhood pixel square of the pixel square comprises: dividing the pixel square in the submarine exploration image; and calculating the pixel difference between the pixel square and the neighborhood pixel square of the pixel square using the following formula:

[0086]

[0087] wherein β represents the pixel difference, x represents a pixel value of a center pixel point in the pixel square, and the pixel mean values of the adjacent pixel points in four directions of the center pixel point in the pixel square.

[0088] In an embodiment of the present application, the construction of the square quadtree of the submarine exploration image based on the pixel difference comprises: when the pixel difference is greater than a preset threshold, taking the center pixel point in the pixel square corresponding to the pixel difference as the center, dividing the submarine exploration image into four grids, and taking the four grids as secondary quadtree nodes; when the size of the four grids is not less than a preset size, performing multi-level grid division on the four grids to obtain multi-level grids, and taking the multi-level grids as multi-level quadtree nodes; when the size of the multi-level grid is less than the preset size, taking the multi-level grid as a quadtree leaf node; and determining the square quadtree based on the center pixel point in the pixel square, the secondary quadtree nodes, the multi-level quadtree nodes and the quadtree leaf node.

[0089] wherein the preset threshold is set to 6, and the preset size is set to 4*4, and can also be set according to actual conditions.

[0090] Referring to FIG. 1, Figure 3 FIG. 1 is a schematic diagram of a square quadtree of a visual-inertial semi-dense reconstruction method provided by an embodiment of the present application. Figure 1 In FIG. 1, each square represents a square. Figure 3

[0091] Optionally, the process of selecting the smallest square from the square quadtree comprises: taking the grid corresponding to the quadtree leaf node in the square quadtree as the smallest square. ​

[0092] S2. Identify the center pixel of the smallest square and construct the epipolar line segment of the center pixel.

[0093] In this embodiment of the invention, an epipolar line segment of the center pixel is constructed for use in an epipolar search method to query the matching point of the center pixel.

[0094] In one embodiment of the present invention, constructing the epipolar segment of the center pixel includes: obtaining a first visual point and a second visual point of the center pixel; taking the position where the distance from the center pixel is the minimum value as the starting point; constructing a depth range segment of the center pixel from the starting point along the direction from the first visual point to the position of the minimum value; identifying the visual interval from the second visual point to the depth range segment; and within the visual interval, querying a straight line segment parallel to the depth range segment, and taking the straight line segment as the epipolar segment.

[0095] See Figure 4 As shown, this is one embodiment of the present invention. Figure 1 A schematic diagram of the epipolar segment is provided for a visual-inertial semi-dense reconstruction method. Figure 4 In the diagram, P1 represents the center pixel, O1 represents the camera's visual angle, O2 represents the IMU's visual angle, d represents the depth range, and l2 represents the epipolar line.

[0096] S3. When the number of pixels contained in the epipolar segment is less than a first preset number, the first matching pixel that matches the center pixel is identified using the preset pyramid optical flow method, and a first matching point pair between the center pixel and the first matching pixel is obtained.

[0097] When the number of pixels contained in the epipolar segment is less than the first preset number, it indicates that the epipolar segment is short. The number of matching points of the center pixel found by the epipolar search method will be less. Therefore, the pyramid optical flow method is used to find the matching points of the center pixel.

[0098] The first preset number is set to 2, but it can also be set according to the actual situation.

[0099] Optionally, the step of identifying the first matching pixel point that matches the center pixel point using a preset pyramid optical flow method includes: first, constructing a pyramid from the original image composed of the center pixel point by continuously reducing the resolution, with each layer being a scaled-down version of the original image; second, in each layer of the pyramid, using some feature point extraction algorithms (such as SIFT, Harris, etc.) to find some stable and representative image feature points; and in the image pyramids of adjacent layers, using some feature point matching algorithms (such as the LK algorithm) to match similar feature point pairs.

[0100] S4, when the number of pixel points contained in the epipolar segment is not less than the first preset number and less than a second preset number, identifying a second matching pixel point matching the center pixel point from the epipolar segment to obtain a second matching point pair between the center pixel point and the second matching pixel point.

[0101] When the number of pixel points contained in the epipolar segment is not less than the first preset number and less than the second preset number, it indicates that the number of pixel points contained in the epipolar segment is moderate, and the matching point of the center pixel point can be queried by the epipolar search method.

[0102] The second preset number is set to 70.

[0103] In an embodiment of the present application, the second matching pixel point matching the center pixel point is identified from the epipolar segment, comprising: collecting the pixel points contained in the epipolar segment; calculating the similarity between the center pixel point and the pixel points contained in the epipolar segment; when the similarity is greater than a preset similarity, the pixel point contained in the epipolar segment corresponding to the similarity is taken as the second matching pixel point.

[0104] The preset similarity is set to 90%.

[0105] S5, when the number of pixel points contained in the epipolar segment is not less than the second preset number, selecting a third matching pixel point matching the center pixel point from the block where the epipolar segment is located to obtain a third matching point pair between the center pixel point and the third matching pixel point.

[0106] When the number of pixel points contained in the epipolar segment is not less than the second preset number, the number of pixel points contained in the epipolar segment is relatively large, and the process of comparing the similarity between each pixel point on the epipolar line and the center pixel point is relatively cumbersome, so the block matching method is used to query the block matching point.

[0107] In an embodiment of the present application, the third matching pixel point matching the center pixel point is selected from the block where the epipolar segment is located, comprising: selecting a first block from the neighborhood of the center pixel point, and extracting a second block with the same size as the first block from the block where the epipolar segment is located; the difference value between the first block and the second block is calculated by the following formula:

[0108] S(A,B) SSD =∑ i,j (A(i,j)-B(i,j)) 2

[0109] Wherein, S(A,B)SSD represents a difference value between the first block and the second block, A(i,j) represents a gray value of the first block minus a mean value of each small block, (i,j) represents a coordinate in the block, and B(i,j) represents a gray value of the second block minus a mean value of each small block;

[0110] When the difference value is less than a preset difference value, a center pixel point in the second block corresponding to the difference value is taken as the third matching pixel point.

[0111] The preset difference value is set to 0.5.

[0112] S6, based on the pixel point motion information of the underwater exploration image in the visual-inertial SLAM system, performing depth analysis on the first matching point pair, the second matching point pair and the third matching point pair to obtain an analysis depth of the center pixel point, and splicing each analysis depth in the analysis depth to obtain a scene semi-dense reconstruction result of the underwater exploration image.

[0113] The embodiment of the present application performs depth analysis on the first matching point pair, the second matching point pair and the third matching point pair based on the pixel point motion information of the underwater exploration image in the visual-inertial SLAM system, which is used to estimate the spatial position of the feature point by using camera motion in order to achieve good semi-dense reconstruction effect on the VI-SLAM system, and with the aid of the front-end IM mismatching elimination algorithm, the visual-inertial SLAM has obtained relatively accurate motion information, so that the matching point pairs obtained by the epipolar search and block matching strategy can be depth estimated by means of these motion information. For the pixels obtained in the monocular system, the depth information of the pixels cannot be solved by simply relying on an image frame, so it is necessary to estimate the observation under different angles, that is, to estimate the depth information by triangulation.

[0114] The analysis depth refers to 3D position information of the center pixel point.

[0115] In an embodiment of the present application, the depth analysis on the first matching point pair, the second matching point pair and the third matching point pair based on the pixel point motion information of the underwater exploration image in the visual-inertial SLAM system to obtain the analysis depth of the center pixel point comprises: calculating the depth interval of the center pixel point by using the following formula:

[0116]

[0117]

[0118] wherein (s1, s2) represents the depth interval, s1 represents a depth interval corresponding to a center pixel in a certain matching point pair of the first matching point pair, the second matching point pair, and the third matching point pair, s2 represents a depth interval corresponding to a matching pixel in the certain matching point pair, R and t constitute a transformation matrix from a world coordinate system to a camera coordinate system, x1 represents a coordinate of the center pixel in the certain matching point pair, and x2 represents a coordinate of the matching pixel in the certain matching point pair;

[0119] Based on the depth interval, the coordinates of the center pixel in the first matching point pair, the second matching point pair, and the third matching point pair are converted to obtain an analysis depth of the center pixel.

[0120] wherein the depth interval has a similar meaning to the depth range segment.

[0121] Referring to FIG. 1, Figure 5 a depth interval of a visual-inertial semi-dense reconstruction method provided in an embodiment of the present application is shown. In the Figure 1 depth interval of P1 and the depth interval of P2 can be used to calculate the coordinates of P, and the coordinates of P are the analysis depth. Figure 5

[0122] Optionally, the process of splicing each analysis depth in the analysis depth is splicing the analysis depth according to a timestamp to obtain a final scene semi-dense reconstruction result.

[0123] ​It can be seen that the embodiment of the application first calculates the pixel difference between the pixel squares in the submarine exploration image and the neighborhood pixel squares of the pixel squares, so as to divide the image frame into blocks using the quadtree strategy, so that the pixels in the same block have similar pixel values, and the pixel values in the leaf nodes, that is, the smallest blocks, are obviously different from other blocks adjacent thereto and have high image gradients, which can meet the requirements of three-dimensional reconstruction. Secondly, the embodiment of the application constructs the epipolar segment of the center pixel point, so as to query the matching point of the center pixel point by using the epipolar search method. Further, the embodiment of the application performs depth analysis on the first matching point pair, the second matching point pair and the third matching point pair based on the motion information of the pixel points in the submarine exploration image in the visual-inertial SLAM system, so as to estimate the spatial position of the feature point by using camera motion in order to achieve good semi-dense reconstruction effect on the VI-SLAM system. With the aid of the front-end IM (inlier-outlier elimination algorithm), the visual-inertial SLAM has obtained relatively accurate motion information, so that the motion information can be used to perform depth estimation on the matching point pairs obtained by the epipolar search and the block matching strategy. For the pixels obtained in the monocular system, the depth information of the pixels cannot be solved by simply relying on one image frame, so it is necessary to estimate the depth information by using the triangulation method under different viewing angles. Therefore, the visual-inertial semi-dense reconstruction method provided by the embodiment of the application can realize the semi-dense scene reconstruction method suitable for deep-sea robots and optimize the old semi-dense scene reconstruction algorithm.

[0124] Embodiment two

[0125] In order to more clearly understand the application, the following further explains the case that the embodiment of the application calculates the pixel difference between the pixel squares in the submarine exploration image and the neighborhood pixel squares of the pixel squares, and constructs the quadtree of the pixel squares in the submarine exploration image based on the pixel difference.

[0126] The application provides a visual-inertial semi-dense reconstruction method, which comprises the following steps:

[0127] The embodiment of the application calculates the pixel difference between the pixel squares in the submarine exploration image and the neighborhood pixel squares of the pixel squares, so as to divide the image frame into blocks using the quadtree strategy, so that the pixels in the same block have similar pixel values, and the pixel values in the leaf nodes, that is, the smallest blocks, are obviously different from other blocks adjacent thereto and have high image gradients, which can meet the requirements of three-dimensional reconstruction.

[0128] In an embodiment of the present application, the calculating the pixel difference between the pixel square in the submarine exploration image and the neighborhood pixel square of the pixel square comprises: dividing the pixel square in the submarine exploration image; calculating the pixel difference between the pixel square and the neighborhood pixel square of the pixel square by using the following formula:

[0129]

[0130] Wherein, β represents the pixel difference, x represents the pixel value of the center pixel point in the pixel square, represents the average pixel value of the adjacent pixel points in the four directions of the center pixel point in the pixel square.

[0131] In an embodiment of the present application, the constructing the square quadtree of the submarine exploration image based on the pixel difference comprises: when the pixel difference is greater than a preset threshold, taking the center pixel point in the pixel square corresponding to the pixel difference as the center, dividing the submarine exploration image into four grids, and taking the four grids as the secondary quadtree nodes; when the size of the four grids is not less than a preset size, performing multi-level grid division on the four grids to obtain multi-level grids, and taking the multi-level grids as multi-level quadtree nodes; when the size of the multi-level grids is less than the preset size, taking the multi-level grids as quadtree leaf nodes; and determining the square quadtree based on the center pixel point in the pixel square, the secondary quadtree nodes, the multi-level quadtree nodes and the quadtree leaf nodes.

[0132] Wherein, the preset threshold is set to 6, and the preset size is set to 4*4, which can also be set according to the actual situation.

[0133] Embodiment three

[0134] As Figure 6 shown in the figure is the functional module diagram of the visual inertial semi-dense reconstruction device of the present application.

[0135] The visual inertial semi-dense reconstruction device 600 of the present application can be installed in an electronic device. According to the realized functions, the visual inertial semi-dense reconstruction device can comprise a square selection module 601, a epipolar line construction module 602, a first identification module 603, a second identification module 604, a third selection module 605 and a depth splicing module 606. The modules of the present application can also be called units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, and are stored in the memory of the electronic device.

[0136] In the embodiments of the present application, the functions of each module / unit are as follows:

[0137] The square selection module 601 is configured to collect a submarine exploration image in a visual front end of a visual-inertial SLAM system, calculate a pixel distance between a pixel square and a neighborhood pixel square of the pixel square in the submarine exploration image, construct a square quad tree of the submarine exploration image based on the pixel distance, and select a minimum square from the square quad tree.

[0138] The epipolar line construction module 602 is configured to identify a center pixel point of the minimum square and construct an epipolar line segment of the center pixel point.

[0139] The first identification module 603 is configured to, when a number of pixel points contained in the epipolar line segment is less than a first preset number, identify a first matching pixel point matched with the center pixel point by using a preset pyramid optical flow method, and obtain a first matching point pair between the center pixel point and the first matching pixel point.

[0140] The second identification module 604 is configured to, when the number of pixel points contained in the epipolar line segment is not less than the first preset number and less than a second preset number, identify a second matching pixel point matched with the center pixel point from the epipolar line segment, and obtain a second matching point pair between the center pixel point and the second matching pixel point.

[0141] The third selection module 605 is configured to, when the number of pixel points contained in the epipolar line segment is not less than the second preset number, select a third matching pixel point matched with the center pixel point from a block in which the epipolar line segment is located, and obtain a third matching point pair between the center pixel point and the third matching pixel point.

[0142] The depth splicing module 606 is configured to perform depth analysis on the first matching point pair, the second matching point pair and the third matching point pair based on pixel point motion information of the submarine exploration image in the visual-inertial SLAM system, obtain an analysis depth of the center pixel point, splice each analysis depth in the analysis depth, and obtain a scene semi-dense reconstruction result of the submarine exploration image.

[0143] In detail, the modules in the visual-inertial semi-dense reconstruction apparatus 600 in the embodiment of the present application use the same technical means as the visual-inertial semi-dense reconstruction method in the above Figure 1 , and can produce the same technical effects, which will not be described here again.

[0144] Embodiment Four

[0145] As Figure 7 shown is a structural schematic diagram of an electronic device for implementing the visual-inertial semi-dense reconstruction method.

[0146] The electronic device can include a processor 70, a memory 71, a communication bus 72, and a communication interface 73, and can further include a computer program stored in the memory 71 and executable on the processor 70, such as a visual-inertial semi-dense reconstruction program.

[0147] The processor 70 can be composed of integrated circuits in some embodiments, for example, can be composed of a single packaged integrated circuit, or can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, combinations of various control chips, etc. The processor 70 is the control core of the electronic device, connects various components of the entire electronic device through various interfaces and lines, executes or runs programs or modules stored in the memory 71 (such as executing a visual-inertial semi-dense reconstruction program, etc.), and calls data stored in the memory 71 to perform various functions of the electronic device and process data.

[0148] The memory 71 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. The memory 71 can be an internal storage unit of the electronic device in some embodiments, for example, a mobile hard disk of the electronic device. The memory 71 can also be an external storage device of the electronic device in other embodiments, for example, a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 71 can include both an internal storage unit and an external storage device of the electronic device. The memory 71 can be used not only to store application software and various data installed on the electronic device, such as the code of a database configuration connection program, but also to temporarily store data that has been output or will be output.

[0149] The communication bus 72 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize the connection and communication between the memory 71 and at least one processor 70, etc.

[0150] The communication interface 73 is used for communication between the electronic device 7 and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface can be a display, an input unit (such as a keyboard), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. Among them, the display can also be appropriately called a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0151] Figure 7 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 7 The structure shown does not constitute a limitation on the electronic device, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0152] For example, although not shown, the electronic device can also include a power supply (such as a battery) to power each component. Preferably, the power supply can be logically connected to the at least one processor 70 through a power management device, so that functions such as charge management, discharge management, and power consumption management can be realized through the power management device. The power supply can also include one or more direct current or alternating current power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described here.

[0153] It should be understood that the embodiments are only for illustration and do not limit the scope of the patent invention.

[0154] The database configuration connection program stored in the memory 71 in the electronic device is a combination of multiple computer programs, which, when running in the processor 70, can realize:

[0155] Collecting a submarine exploration image in a visual front end of a visual-inertial SLAM system, calculating a pixel difference between a pixel square and a neighborhood pixel square of the pixel square in the submarine exploration image, constructing a quad tree of the pixel square of the submarine exploration image based on the pixel difference, and selecting a minimum square from the quad tree of the pixel square;

[0156] identify a center pixel of the minimum square, construct an epipolar line segment of the center pixel;

[0157] when the number of pixel points contained in the epipolar line segment is less than a first preset number, identify a first matching pixel point matched with the center pixel by using a preset pyramid optical flow method, and obtain a first matching point pair between the center pixel and the first matching pixel point;

[0158] when the number of pixel points contained in the epipolar line segment is not less than the first preset number and less than a second preset number, identify a second matching pixel point matched with the center pixel from the epipolar line segment, and obtain a second matching point pair between the center pixel and the second matching pixel point;

[0159] when the number of pixel points contained in the epipolar line segment is not less than the second preset number, select a third matching pixel point matched with the center pixel from a square in which the epipolar line segment is located, and obtain a third matching point pair between the center pixel and the third matching pixel point;

[0160] based on the motion information of the pixel points of the submarine exploration image in the visual-inertial SLAM system, perform depth analysis on the first matching point pair, the second matching point pair and the third matching point pair to obtain an analysis depth of the center pixel, and splice each analysis depth in the analysis depth to obtain a scene semi-dense reconstruction result of the submarine exploration image.

[0161] Specifically, the specific implementation method of the processor 70 to the above computer program can refer to Figure 1 The description of related steps in the corresponding embodiments will not be repeated here.

[0162] Further, the modules / units integrated in the electronic device, if realized in the form of software function units and sold or used as independent products, can be stored in a nonvolatile computer readable storage medium. The storage medium can be volatile or nonvolatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM).

[0163] The application also provides a storage medium, the readable storage medium stores a computer program, and the computer program can realize the following when executed by a processor of an electronic device:

[0164] Collecting a submarine exploration image in a vision front end of a visual-inertial SLAM system, calculating a pixel distance between a pixel square and a neighborhood pixel square of the pixel square in the submarine exploration image, constructing a quadtree of the pixel square of the submarine exploration image based on the pixel distance, and selecting a minimum pixel square from the quadtree;

[0165] Identifying a center pixel point of the minimum pixel square, and constructing a epipolar segment of the center pixel point;

[0166] When a number of pixel points contained in the epipolar segment is less than a first preset number, identifying a first matching pixel point matched with the center pixel point by using a preset pyramid optical flow method, and obtaining a first matching point pair between the center pixel point and the first matching pixel point;

[0167] When the number of pixel points contained in the epipolar segment is not less than the first preset number and less than a second preset number, identifying a second matching pixel point matched with the center pixel point from the epipolar segment, and obtaining a second matching point pair between the center pixel point and the second matching pixel point;

[0168] When the number of pixel points contained in the epipolar segment is not less than the second preset number, selecting a third matching pixel point matched with the center pixel point from a block where the epipolar segment is located, and obtaining a third matching point pair between the center pixel point and the third matching pixel point;

[0169] Based on pixel point motion information of the submarine exploration image in the visual-inertial SLAM system, performing depth analysis on the first matching point pair, the second matching point pair and the third matching point pair to obtain an analysis depth of the center pixel point, and splicing each analysis depth in the analysis depth to obtain a scene semi-dense reconstruction result of the submarine exploration image.

[0170] In several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the above-described device embodiments are merely illustrative, and the division of the modules is merely a logical function division, and there can be another division manner in actual implementation.

[0171] The modules described as separated components can or can not be physically separated, and the components displayed as modules can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments.

[0172] In addition, each of the functional modules in various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of hardware plus software function modules.

[0173] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be realized in other specific forms without departing from the spirit or essential characteristics of the present application.

[0174] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any additional reference signs in the claims should not be considered as limiting the claims involved.

[0175] It should be noted that, in this document, relational terms such as "first" and "second", and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element preceded by "comprises... " does not, without more limitations, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the recited element.

[0176] The above description is merely one specific implementation of the present application, enabling one skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the present application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual-inertial semi-dense reconstruction method, characterized in that, The method comprises the following steps: Collecting a submarine exploration image in a visual front end of a visual-inertial SLAM system, calculating a pixel difference between a pixel square and a neighborhood pixel square of the pixel square in the submarine exploration image, constructing a quadtree of the pixel square based on the pixel difference, and selecting a minimum pixel square from the quadtree; Identifying a center pixel point of the minimum pixel square and constructing a epipolar segment of the center pixel point; When the number of pixel points contained in the epipolar segment is less than a first preset number, identifying a first matching pixel point matched with the center pixel point by using a preset pyramid optical flow method to obtain a first matching point pair between the center pixel point and the first matching pixel point; When the number of pixel points contained in the epipolar segment is not less than the first preset number and less than a second preset number, identifying a second matching pixel point matched with the center pixel point from the epipolar segment to obtain a second matching point pair between the center pixel point and the second matching pixel point; When the number of pixel points contained in the epipolar segment is not less than the second preset number, selecting a third matching pixel point matched with the center pixel point from a block where the epipolar segment is located to obtain a third matching point pair between the center pixel point and the third matching pixel point; Based on the pixel point motion information of the submarine exploration image in the visual-inertial SLAM system, performing depth analysis on the first matching point pair, the second matching point pair and the third matching point pair to obtain an analysis depth of the center pixel point, and splicing each analysis depth in the analysis depth to obtain a scene semi-dense reconstruction result of the submarine exploration image; The depth analysis on the first matching point pair, the second matching point pair and the third matching point pair based on the pixel point motion information of the submarine exploration image in the visual-inertial SLAM system to obtain the analysis depth of the center pixel point comprises: The depth interval of the center pixel point is calculated by using the following formula: wherein, denotes the depth interval of the center pixel in the certain matching point pair of the first matching point pair, the second matching point pair, the third matching point pair, denotes the depth interval of the center pixel in the certain matching point pair of the first matching point pair, the second matching point pair, the third matching point pair, denotes the depth interval of the matching pixel in the certain matching point pair of the first matching point pair, the second matching point pair, the third matching point pair, and denotes the transformation matrix from the world coordinate system to the camera coordinate system, denotes the coordinate of the center pixel in the certain matching point pair of the first matching point pair, the second matching point pair, the third matching point pair, denotes the coordinate of the matching pixel in the certain matching point pair of the first matching point pair, the second matching point pair, the third matching point pair. Based on the depth interval, the coordinates of the center pixel points in the first matching point pair, the second matching point pair and the third matching point pair are converted to obtain the analysis depth of the center pixel point.

2. The visual-inertial semi-dense reconstruction method of claim 1, wherein, The calculation of the pixel difference between the pixel square and the neighborhood pixel square of the pixel square in the submarine exploration image comprises: Dividing the pixel square in the submarine exploration image; The pixel difference between the pixel square and the neighborhood pixel square of the pixel square is calculated by using the following formula: wherein, represents the pixel distance of the pixel square, represents the pixel value of the center pixel point in the pixel square, represents the pixel mean value of the neighboring pixel points in the four directions of the center pixel point in the pixel square.

3. The visual-inertial semi-dense reconstruction method of claim 1, wherein, The construction of the quadtree of the submarine exploration image based on the pixel difference comprises: When the pixel difference is greater than a preset threshold, the submarine exploration image is divided into four grids with the center pixel point in the pixel square corresponding to the pixel difference as the center, and the four grids are taken as secondary quadtree nodes; When the size of the four grids is not less than a preset size, the four grids are subjected to multi-level grid division to obtain multi-level grids, and the multi-level grids are taken as multi-level quadtree nodes; when the size of the multi-level grid is less than the preset size, the multi-level grid is taken as a quadtree leaf node; determine the square quadtree based on the center pixel point in the pixel square, the second-level quadtree node, the multi-level quadtree node and the quadtree leaf node.

4. The visual-inertial semi-dense reconstruction method of claim 1, wherein, the constructing the epipolar line segment of the center pixel point comprises: acquire a first visual point and a second visual point of the center pixel point; take the position with the minimum distance from the center pixel point as a starting point; construct a depth range segment of the center pixel point in the direction from the starting point to the position with the minimum distance along the first visual point; identify a visual interval of the second visual point to the depth range segment; in the visual interval, query a straight line segment parallel to the depth range segment, and take the straight line segment as the epipolar line segment.

5. The visual-inertial semi-dense reconstruction method of claim 1, wherein, the identifying a second matching pixel point matching the center pixel point from the epipolar line segment comprises: collect the pixel points contained in the epipolar line segment; calculate the similarity degree between the center pixel point and the pixel points contained in the epipolar line segment; when the similarity degree is greater than a preset similarity degree, take the pixel point contained in the epipolar line segment corresponding to the similarity degree as the second matching pixel point.

6. The visual-inertial semi-dense reconstruction method of claim 1, wherein, the selecting a third matching pixel point matching the center pixel point from the square where the epipolar line segment is located comprises: select a first square from the neighborhood of the center pixel point, and extract a second square with the same size as the first square from the square where the epipolar line segment is located; calculate the difference value between the first square and the second square by using the following formula: wherein, represents a difference value between the first patch and the second patch, represents a gray value of the first patch minus the mean value of each small patch, represents a coordinate in the patch, represents a gray value of the second patch minus the mean value of each small patch; when the difference value is less than a preset difference value, take the center pixel point in the second square corresponding to the difference value as the third matching pixel point.

7. A visual-inertial semi-dense reconstruction apparatus for implementing the visual-inertial semi-dense reconstruction method of any one of claims 1-6, characterized in that, the device comprises: a square selecting module, configured to collect a submarine exploration image in a visual front end of a visual-inertial simultaneous localization and mapping (SLAM) system, calculate a pixel difference between a pixel square in the submarine exploration image and a neighborhood pixel square of the pixel square, construct a square quadtree of the submarine exploration image based on the pixel difference, and select a minimum square from the square quadtree; an epipolar line constructing module, configured to identify a center pixel point of the minimum square, and construct an epipolar line segment of the center pixel point; a first identifying module, configured to, when the number of pixel points contained in the epipolar line segment is less than a first preset number, identify a first matching pixel point matching the center pixel point by using a preset pyramid optical flow method, and obtain a first matching point pair between the center pixel point and the first matching pixel point; a second identifying module, configured to, when the number of pixel points contained in the epipolar line segment is not less than the first preset number and less than a second preset number, identify a second matching pixel point matching the center pixel point from the epipolar line segment, and obtain a second matching point pair between the center pixel point and the second matching pixel point; The third selecting module is configured to select a third matching pixel point matched with the center pixel point from a block where the epipolar segment is located when the number of pixel points contained in the epipolar segment is not less than the second preset number, and obtain a third matching point pair between the center pixel point and the third matching pixel point. The depth splicing module is configured to perform depth analysis on the first matching point pair, the second matching point pair and the third matching point pair based on pixel point motion information of the underwater exploration image in the visual-inertial SLAM system, obtain an analysis depth of the center pixel point, splice each analysis depth in the analysis depth, and obtain a scene semi-dense reconstruction result of the underwater exploration image.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the visual-inertial semi-dense reconstruction method according to any one of claims 1 to 6.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the visual-inertial semi-dense reconstruction method according to any one of claims 1 to 6.

Images