Visual odometer method, system and equipment based on point-line characteristics and medium

Abstract

The invention discloses a visual odometer method, system and device based on point-line features and a medium. Visual odometers are crucial to navigation and planning of autonomous robots, but weak light conditions, violent illumination changes and low texture scenes make a significant challenge to odometer estimation. Therefore, according to the visual odometer method based on the point-line features, the point-line features and deep learning are integrated, the point features and the line features are obtained through a self-supervised learning model for interest point detection and a line detection mode combining line light flow tracking and cross constraint, and the influence of complex lighting conditions is overcome. Meanwhile, the key frame is selected based on point feature counting and line feature overlapping angle constraint, and sparse matching is performed by integrating the convolutional neural network and the graph neural network, so that the accuracy and the calculation efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, system, device and medium for visual odometer based on point and line features. Background Art

[0002] Front-end visual odometry (VO) methods estimate the camera motion trajectory through continuous images acquired by the camera. It is a key technology in fields such as autonomous driving and robot navigation. Traditional VO methods are mainly divided into feature-based methods and direct methods. Feature-based methods usually rely on the extraction and matching of feature points, such as ORB, SIFT, and SURF. They estimate camera motion by tracking feature points in adjacent frames using triangulation or PnP algorithms. However, these methods are sensitive to changes in environmental illumination and texture. For visually unfriendly scenes (such as low light, drastic changes in illumination, and weak texture scenes), they may not be able to recognize and associate feature information in the current environment, resulting in unstable performance in complex and changing scenes, and may even cause tracking loss. Direct rules estimate motion by directly minimizing the difference in image pixel intensity, such as direct sparse odometry (DSO). These methods rely less on feature points, but have high requirements for calculation accuracy and are easily affected by camera noise and occlusion.

[0003] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a visual odometer method, system, device and medium based on point and line features in view of the deficiencies in the prior art.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a visual odometer method based on point and line features, wherein the visual odometer method based on point and line features specifically includes: Obtaining a sequence of images of the object; Performing point feature extraction and line feature extraction on each image frame in the image sequence respectively to obtain point features and line features of each image frame, wherein the line feature extraction adopts a line optical flow tracking and cross constraint method; Selecting key frames in the image sequence based on point and line features and line features; The point features and line features of the selected key frames are matched through a preset depth matching model to obtain matching features, and the pose information of the photographed object is estimated based on the matching features.

[0006] The visual odometer method based on point and line features, wherein the extraction process of the line feature extraction specifically includes: Based on the grayscale invariance criterion, candidate line features are selected in the image frame through optical flow tracking; The selected candidate line features are screened using the angle between each candidate line feature and other candidate line features as a cross constraint to extract the line features of the image frame.

[0007] The visual odometer method based on point and line features, wherein the cross constraint is: < , , in, represents the angle between a candidate line feature and a candidate line feature, represents the angle between the candidate line feature and another candidate line feature, represents the difference between two angles, Indicates the preset angle threshold.

[0008] The visual odometer method based on point and line features, wherein the extraction process of the point feature extraction specifically includes: Inputting the image frame into an encoder in a preset feature point detector, and converting the image frame into a tensor through the encoder; The tensor is input into a decoder in the feature point detector, the response of the feature point is extracted by the decoder, and the point features of the image frame are determined based on the response of the feature point.

[0009] In the visual odometer method based on point and line features, the preset feature point detector adopts a SuperPoint framework, and the SuperPoint framework adjusts the score threshold of the feature point through an adaptive threshold mechanism.

[0010] The visual odometer method based on point and line features, wherein the step of selecting key frames in the image sequence based on point and line features and line features is as follows: Based on the point features and the line features, key frames are selected in the image sequence according to a preset selection strategy, wherein the selection strategy includes a feature quality-based strategy, a time interval-based strategy, and an overlap angle constraint-based strategy.

[0011] The visual odometer method based on point and line features, wherein the overlapping angle constraint strategy is: If the state difference between the line feature overlapping angle state corresponding to the selected target key frame and the line feature overlapping angle state corresponding to the candidate image frame is less than the preset angle threshold, the candidate image frame is selected as the key frame, wherein the target key frame is the last key frame in the selected key frames in chronological order.

[0012] The second aspect of the present application provides a visual odometer system based on point and line features, wherein the visual odometer system based on point and line features specifically includes: An acquisition module, used for acquiring an image sequence of a photographed object; A feature extraction module, used for performing point feature extraction and line feature extraction on each image frame in the image sequence to obtain point features and line features of each image frame, wherein the line feature extraction adopts a line optical flow tracking and cross constraint method; A key frame selection module, used for selecting key frames in the image sequence based on point line features and line features; The deep feature matching module is used to perform feature matching on the point features and line features of the selected key frames through a preset deep matching model to obtain matching features, and estimate the pose information of the photographed object based on the matching features.

[0013] A third aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the visual odometer methods based on point and line features as described above.

[0014] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in any of the above-mentioned visual odometer methods based on point and line features are implemented.

[0015] Beneficial effects: 1) This application extracts line features through optical flow tracking, which overcomes the problem of feature extraction in low-light environments and areas with weak textures, thereby improving the accuracy of feature matching in low-texture and fast-motion scenes, and further improving the accuracy of posture estimation.

[0016] 2) This application introduces an adaptive threshold mechanism to dynamically adjust the threshold of feature points according to different scene changes, thereby improving the accuracy and robustness of feature extraction in complex environments.

[0017] 3) This application takes into account the angle changes of line features by adding an overlapping angle constraint strategy, so that key frames can be selected more accurately when the scene structure changes greatly, thereby improving the stability and accuracy of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a visual odometer method based on point and line features provided in an embodiment of the present application.

[0020] Figure 2 A flowchart schematic of an example of a visual odometer method based on point and line features provided in an embodiment of the present application.

[0021] Figure 3 Schematic diagram of the correspondence between line features between two adjacent frames of images.

[0022] Figure 4 A schematic diagram of the angle between two line features.

[0023] Figure 5 Another illustration of the angle between two line features.

[0024] Figure 6 This is a comparison chart of the effects of the line features extracted by the embodiment of the present application and the line features extracted by the LSD method in the prior art.

[0025] Figure 7 This is the principle flow chart of the feature point detector.

[0026] Figure 8 This is the principle flow chart of the deep matching model.

[0027] Fig. 9 A block diagram of the principle of a visual odometer system based on point and line features provided in an embodiment of the present application.

[0028] Fig.10 A functional block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application provide a method, system, device and medium for visual odometer based on point and line features. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0030] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0031] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0032] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0033] After research, it was found that the front-end visual odometry (VO) method estimates the camera motion trajectory through continuous images obtained by the camera. It is a key technology in fields such as autonomous driving and robot navigation. Traditional VO methods are mainly divided into feature-based methods and direct methods. Among them, feature-based methods usually rely on the extraction and matching of feature points, such as ORB, SIFT and SURF, and estimate camera motion by tracking feature points in adjacent frames using triangulation or PnP algorithms. However, these methods are sensitive to changes in environmental illumination and texture. For visually unfriendly scenes (such as low light, drastic changes in illumination, and weak texture scenes), they may not be able to recognize and associate feature information in the current environment, resulting in unstable performance in complex and changing scenes, and even possible tracking loss. Direct rules estimate motion by directly minimizing the difference in image pixel intensity, such as direct sparse odometry (DSO). These methods rely less on feature points, but have high requirements for calculation accuracy and are easily affected by camera noise and occlusion.

[0034] In order to solve the above problems, in an embodiment of the present application, an image sequence of the photographed object is obtained; point feature extraction and line feature extraction are performed on each image frame in the image sequence respectively to obtain point features and line features of each image frame, wherein the line feature extraction adopts the method of line optical flow tracking and cross constraint; key frames are selected in the image sequence based on point line features and line features; the point features and line features of the selected key frames are feature matched through a preset deep matching model to obtain matching features, and the pose information of the photographed object is estimated based on the matching features. The embodiment of the present application extracts line features through optical flow tracking, which overcomes the feature extraction in low light environments and areas with weak textures, thereby improving the accuracy of feature matching in low texture and fast motion scenes, and further improving the accuracy of pose estimation.

[0035] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0036] This embodiment provides a visual odometer method based on point and line features, such as Figure 1 and Figure 2 As shown, the method includes: S10: Acquire an image sequence of the object.

[0037] Specifically, the image sequence may be obtained by capturing images of the photographed object through an image acquisition device, wherein the image acquisition device may be an RGB-D camera mounted on a carrier (such as a robot, a drone, etc.), and a depth profile image may be obtained by using the RGB-D camera using a depth profile estimation method. The image sequence may include system image frames acquired sequentially and continuously from different times and different orientations of the photographed object.

[0038] In one implementation, the image acquisition device for acquiring an image sequence of a photographed object is a binocular camera, through which left and right eye images can be obtained respectively, and a depth image is obtained by using stereo vision technology to calculate the distance of pixels through the left and right eye parallax, and the depth image contains the three-dimensional world coordinate information of all pixels. In other words, each image frame in the image sequence includes a left eye image and a right eye image. Of course, in practical applications, a monocular camera can also be used to acquire the photographed object to form an image sequence, etc.

[0039] S20, performing point feature extraction and line feature extraction on each image frame in the image sequence respectively to obtain point features and line features of each image frame.

[0040] Specifically, point feature extraction is used to extract point features of an image, and line feature extraction is used to extract line features of an image. That is, point features and line features of each image frame can be extracted through point feature extraction and line feature extraction, that is, point features and line features of each image frame can be obtained.

[0041] In one implementation, the line feature extraction adopts line optical flow tracking and cross constraint, optimizes abnormal line segments through optical flow tracking, and removes messy line features in complex scenes through cross constraint to extract accurate line features. Based on this, the extraction process of the line feature extraction specifically includes: Based on the grayscale invariance criterion, candidate line features are selected in the image frame through optical flow tracking; The selected candidate line features are screened using the angle between each candidate line feature and other candidate line features as a cross constraint to extract the line features of the image frame.

[0042] Specifically, the candidate line features are selected in the image frame by optical flow tracking. In the process of optical flow tracking, the relationship between pixel intensity and motion can be expressed as: , in, and represents the optical flow vector, Indicates that the image frame is The gray gradient, Indicates that the image frame is The gray gradient in the direction, Represents the change of image grayscale over time, that is, The gradient of the direction, express t Moment image, express Moment image.

[0043] The matrix form of the relationship between pixel intensity and motion can be expressed as: .

[0044] In addition, the points on the line segment need to satisfy the collinearity constraint. To this end, when the point set on the line segment is expressed as: , in, and Represent the starting point and ending point of the line segment respectively. represents a positive integer, express and When the Euclidean distance between the two consecutive frames is , the relationship between the corresponding points of the line segments in two consecutive frames of images satisfies: , in, represents the position of the line segment in the next frame, , ,and Respectively represent the change of the starting point position and the change around the starting point The rotation change of Indicates the angle between the line segment and the horizontal direction in the current frame, ( , ) indicates that the next frame is The corresponding points, represents a positive integer, <= .

[0045] In addition, due to the is a minor change, and , thus, if Figure 3 As shown, and( , ) can be expressed as: , Based on the above relationship, the candidate line features are determined and then filtered through the cross constraint, where the cross constraint can be expressed as: < , , Among them, Figure 4 and Figure 5 As shown, represents the angle between the candidate line feature a and the candidate line feature b, represents the angle between the candidate line feature a and another candidate line feature c, represents the difference between two angles, Indicates the preset angle threshold.

[0046] It can be seen that for a candidate line feature, when there are two candidate line features among other candidate line features, and the difference between the angles between the two candidate line features and the candidate line feature is less than the preset angle threshold, then the candidate line feature is in a false detection state, and the candidate line feature is removed to remove the false detection line feature, thereby improving the accuracy of the extracted line feature. Specifically, Figure 6 The effect comparison diagram of the line features extracted by the embodiment of the present application and the line features extracted by the LSD method in the prior art is shown. The accuracy of the line features provided by the embodiment of the present application is higher than that of the line features extracted by the LSD method.

[0047] The embodiment of the present application improves the line detection model by introducing optical flow tracking and cross constraint methods, and then extracts line features through the line detection model. This can overcome the influence of low texture and motion blur, improve the accuracy of the extracted line features, and further improve the accuracy of subsequent posture estimation.

[0048] In one implementation, the point feature extraction process specifically includes: Inputting the image frame into an encoder in a preset feature point detector, and converting the image frame into a tensor through the encoder; The tensor is input into a decoder in the feature point detector, the response of the feature point is extracted by the decoder, and the point features of the image frame are determined based on the response of the feature point.

[0049] Specifically, the preset feature point detector adopts the SuperPoint framework. SuperPoint adjusts the score threshold of the feature point through an adaptive threshold mechanism, and extracts point features based on the score threshold. The preset feature point detector can be suitable for different application scenarios (for example, normal light scenes and dim light scenes, etc.). Among them, when the preset feature point detector adjusts the score threshold of the feature point through the adaptive threshold mechanism, it will obtain the distribution and matching of the feature points in the image frame, and then dynamically adjust the score threshold based on the distribution and matching, so that the preset feature point detector and descriptor can be sequenced through self-supervision and adapt to different conditions by dynamically adjusting the threshold.

[0050] like Figure 7 As shown, the preset feature point detector includes an encoder and a decoder, the encoder is used to encode the image frame to convert the image frame into a tensor, and the decoder is used to extract the response of the feature point based on the tensor and determine the point feature of the image frame based on the response. Specifically, the image frame is converted into a smaller tensor ,in, and are the width and height of the encoded image, respectively. and are the width and height of the image frame, respectively. , Next, the tensor is fed into the decoder to detect feature points The decoder uses convolutional layers to extract the response of feature points. , the response contains 65 channels, one of which is the "no feature point" channel, and the other 64 channels represent the image frame Then, through the channel-level softmax operation, the "no feature point" channel is removed, and finally the reshape operation is used to convert Convert to the original image frame size , extract the point features of the image frame.

[0051] Furthermore, the loss function used by the preset feature point detector during training is It is calculated by convolution cross entropy loss. The loss function calculation formula can be: , in, Represents the feature point set output by the preset feature point detector, represents an output feature point, represents the real feature point set, Represents a real feature point.

[0052] S30: Select key frames in the image sequence based on point and line features and line features.

[0053] Specifically, a key frame is an image frame of an image sequence, and the key frame may be one or more. That is, one or more key frames may be selected from the image sequence. The key frame may be selected based on one or more of a feature quality strategy, a time interval strategy, and an overlap angle strategy.

[0054] Furthermore, before selecting key frames in the image sequence based on point-line features and line features, the point features and line features can be first superimposed into point-line features, and then key frames can be selected in the image sequence based on the point-line features, wherein the point features and line features are superimposed as superimposing the point features and line features on a feature map, that is, a blank image with the same image size as the image frame can be constructed, and then the point features and line features are respectively mapped to the blank image to obtain point-line features. Of course, in practical applications, other methods can also be used, for example, mapping point features into line features, or mapping line features into point features, etc.

[0055] In one implementation, the step of selecting key frames in the image sequence based on point and line features is as follows: Based on the point features and the line features, key frames are selected from the image sequence according to a preset selection strategy.

[0056] Specifically, the preset selection strategies include a feature quality-based strategy, a time interval-based strategy, and an overlapping angle constraint-based strategy, wherein the feature quality-based strategy is specifically based on the number of currently tracked points and the tracking quality as constraints. Specifically, when the number of currently tracked points is less than a preset threshold, or the tracking quality is reduced, a new keyframe needs to be selected. This is because when the number of currently tracked points is greater than the preset threshold, or the tracking quality is reduced, it indicates that the camera market has changed significantly, and in order to maintain the stability of the system, especially in rapid motion or environmental changes, a new keyframe needs to be selected. In addition, it should be noted that the number of currently tracked points and the tracking quality are determined based on the point features and the line features, and the determination process can adopt existing technologies, which will not be described in detail here.

[0057] The time interval strategy is based on the interval frame number as a constraint condition. Specifically, when the interval frame number exceeds the preset maximum limit time, or at least the minimum number of frames has passed and the mapping thread is in the spatial state, a new key frame needs to be selected.

[0058] Based on the overlapping angle constraint strategy, the overlapping angle state of the line feature is used as a constraint condition. Specifically, when the state difference between the overlapping angle state of the line feature corresponding to the selected target key frame and the overlapping angle state of the line feature corresponding to the candidate image frame is less than the preset angle threshold, the candidate image frame is selected as the key frame, wherein the target key frame is the last key frame in the selected key frames in time order, and the state difference can be expressed as: = - , , , in, Indicates the state difference, Indicates the line feature overlap angle state of the target keyframe, Indicates the target keyframe Overlap angle values; Indicates the line feature overlap angle status of the candidate image frame, Represents the candidate image pair The overlap angle value refers to the angle between the two antenna features.

[0059] The overlapping angle constraint strategy takes into account the camera's motion and the change of the scene structure. When the state difference between the overlapping angle states of two line features is less than the preset angle threshold, it indicates that the surrounding scene structure has changed significantly, and a new keyframe needs to be selected. This application implements the combination of the overlapping angle constraint strategy with the feature quality strategy and the time interval strategy, fully considering the change of the camera field and improving the accuracy of keyframe selection.

[0060] S40, performing feature matching on the point features and line features of the selected key frame through a preset depth matching model to obtain matching features, and estimating the position and posture information of the photographed object based on the matching features.

[0061] Specifically, the matching features are obtained by matching the point features and line features of each key frame, that is, matching the point and line features of each key frame (obtained by superimposing the point features and line features), and the matching features are used for posture information. In other words, the matching features of the present application are integrated with the point and line features, which can improve the feature information carried by the matching features, and further improve the accuracy of the posture information of the photographed object estimated based on the matching features.

[0062] The task of feature matching is to predict the corresponding matching matrix , which corresponds to finding a soft partial assignment matrix . Representation diagram No. Feature points match the above image No. The possibility of feature points. "Soft assignment" means that the matrix elements are not just absolute 0 or 1. The sum of any row or column of is equal to 1. However, in reality, due to factors such as field of view change / occlusion / lighting, The characteristic points in the figure If there is no corresponding matching point, the corresponding sum is not equal to 1. Soft part matching matrix It can be expressed as: , In one implementation, when performing feature matching, the LightGlue model can be used. LightGlue is a flexible image matching library that can calculate the best image fusion strategy based on the feature point information provided by Superpoint, thereby achieving smooth transition and eliminating discontinuities in overlapping areas. Specifically, Figure 8As shown in the figure, the LightGlue model includes L identical network layers, and L network layers jointly process feature matching. Each network layer includes a self-attention unit and a cross-attention unit, and the feature representation is updated by the self-attention unit and the cross-attention unit. In addition, each network layer is connected to a classifier, which will classify based on the updated feature representation of the network layer to decide whether to stop the reasoning process. When a classifier decides to stop the reasoning process, the subsequent network layer will no longer be executed, which can avoid unnecessary calculations and reduce the amount of calculation. In addition, each network layer is connected to a lightweight head. When the classification head corresponding to the network layer decides to stop the reasoning process, the network layer inputs the updated feature representation into the lightweight head, and calculates the feature matching in the feature representation by inputting the lightweight head to obtain the matching feature.

[0063] In summary, the present embodiment provides a visual odometer method based on point and line features, the method comprising acquiring an image sequence of a photographed object; performing point feature extraction and line feature extraction on each image frame in the image sequence respectively to obtain point features and line features of each image frame, wherein the line feature extraction adopts line optical flow tracking and cross constraints; selecting key frames in the image sequence based on point and line features and line features; performing feature matching on the point features and line features of the selected key frames through a preset deep matching model to obtain matching features, and estimating the pose information of the photographed object based on the matching features. The present embodiment of the application extracts line features through optical flow tracking, which overcomes the problem of feature extraction in low-light environments and areas with weak textures, thereby improving the accuracy of feature matching in low-texture and fast-motion scenes, and further improving the accuracy of pose estimation.

[0064] Based on the above-mentioned visual odometer method based on point and line features, this embodiment provides a visual odometer system based on point and line features, such as Fig. 9 As shown, the visual odometer system based on point and line features specifically includes: An acquisition module 100 is used to acquire an image sequence of a photographed object; A feature extraction module 200 is used to perform point feature extraction and line feature extraction on each image frame in the image sequence to obtain point features and line features of each image frame, wherein the line feature extraction adopts a line optical flow tracking and cross constraint method; A key frame selection module 300, used for selecting key frames in the image sequence based on point line features and line features; The depth feature matching module 400 is used to perform feature matching on the point features and line features of the selected key frames through a preset depth matching model to obtain matching features, and estimate the pose information of the photographed object based on the matching features.

[0065] Based on the above-mentioned visual odometer method based on point and line features, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the visual odometer method based on point and line features as described in the above embodiment.

[0066] Based on the above-mentioned visual odometer method based on point and line features, the present application also provides a terminal device, such as Fig.10 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, the display screen 21, the memory 22, and the communications interface 23 may communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communications interface 23 may transmit information. The processor 20 may call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0067] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0068] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0069] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0070] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A visual odometer method based on point and line features, characterized in that: The visual odometer method based on point and line features specifically includes: Obtaining a sequence of images of the object; Performing point feature extraction and line feature extraction on each image frame in the image sequence respectively to obtain point features and line features of each image frame, wherein the line feature extraction adopts a line optical flow tracking and cross constraint method; Selecting key frames in the image sequence based on point and line features and line features; The point features and line features of the selected key frames are matched through a preset depth matching model to obtain matching features, and the pose information of the photographed object is estimated based on the matching features.

2. The visual odometer method based on point and line features according to claim 1, characterized in that: The extraction process of the line feature extraction specifically includes: Based on the grayscale invariance criterion, candidate line features are selected in the image frame through optical flow tracking; The selected candidate line features are screened using the angle between each candidate line feature and other candidate line features as a cross constraint to extract the line features of the image frame.

3. The visual odometer method based on point and line features according to claim 2, characterized in that: The cross constraints are: < ， ， in, represents the angle between a candidate line feature and a candidate line feature, represents the angle between the candidate line feature and another candidate line feature, represents the difference between two angles, Indicates the preset angle threshold.

4. The visual odometer method based on point and line features according to claim 1, characterized in that: The extraction process of the point feature extraction specifically includes: Inputting the image frame into an encoder in a preset feature point detector, and converting the image frame into a tensor through the encoder; The tensor is input into a decoder in the feature point detector, the response of the feature point is extracted by the decoder, and the point features of the image frame are determined based on the response of the feature point.

5. The visual odometer method based on point and line features according to claim 4, characterized in that: The preset feature point detector adopts a SuperPoint framework, and the SuperPoint framework adjusts the score threshold of the feature point through an adaptive threshold mechanism.

6. The visual odometer method based on point and line features according to claim 1, characterized in that: The step of selecting key frames in the image sequence based on point and line features and line features is as follows: Based on the point features and the line features, key frames are selected in the image sequence according to a preset selection strategy, wherein the selection strategy includes a feature quality-based strategy, a time interval-based strategy, and an overlap angle constraint-based strategy.

7. The visual odometer method based on point and line features according to claim 6, characterized in that: The overlapping angle constraint strategy is: If the state difference between the line feature overlapping angle state corresponding to the selected target key frame and the line feature overlapping angle state corresponding to the candidate image frame is less than the preset angle threshold, the candidate image frame is selected as the key frame, wherein the target key frame is the last key frame in the selected key frames in chronological order.

8. A visual odometer system based on point and line features, characterized in that: The visual odometer system based on point and line features specifically includes: An acquisition module, used for acquiring an image sequence of a photographed object; A feature extraction module, used for performing point feature extraction and line feature extraction on each image frame in the image sequence to obtain point features and line features of each image frame, wherein the line feature extraction adopts a line optical flow tracking and cross constraint method; A key frame selection module, used for selecting key frames in the image sequence based on point line features and line features; The deep feature matching module is used to perform feature matching on the point features and line features of the selected key frames through a preset deep matching model to obtain matching features, and estimate the pose information of the photographed object based on the matching features.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the visual odometer method based on point and line features as described in any one of claims 1-7.

10. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the visual odometer method based on point and line features as described in any one of claims 1 to 7 are implemented.

Images