Visual inertial odometer in dynamic environment and design method and system thereof

By combining object detection and MobileSAM technology visual inertial odometry methods, the accuracy problem of SLAM algorithm in distinguishing dynamic objects and static objects in dynamic environments is solved, and high-precision positioning and robustness in different environments is achieved, reducing the Training cost, strong generalization ability.

CN118936507APending Publication Date: 2024-11-12WUHAN UNIV
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410964040.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In dynamic environments, existing SLAM algorithms are difficult to accurately distinguish dynamic objects from static objects, resulting in reduced positioning accuracy and unstable system reliability. The existing methods require specific model training for different environments, and have poor versatility.

Method used

Using a visual inertial odometer design method combining object detection technology and MobileSAM technology, the positioning and camera pose estimation are performed by identifying visual information, performing semantic segmentation, feature point extraction and optical flow tracking, combined with IMU pre-integration, using Hungarian algorithms and Kalman Filtering performs accuracy compensation, generates an image mask to remove dynamic areas, ensuring the universality and robustness of the system in different environments.

Benefits of technology

It realizes a high-precision visual inertial odometer in dynamic environments, improves the accuracy and robustness of the system, reduces the training cost when used in different environments, and has strong generalization ability and expansion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118936507A_ABST
    Figure CN118936507A_ABST
Patent Text Reader

Abstract

The invention discloses a visual inertial odometer in a dynamic environment and a design method and system thereof. The method comprises the following steps: acquiring visual information in the environment; identifying the visual information, and transmitting an identification result as an interactive cue word; according to the interactive cue word, performing semantic segmentation of a specific region by using a Mobile SAM model to obtain a potential high-possibility dynamic region in the current image frame; according to the potential high-possibility dynamic region, feature point extraction and tracking are carried out in combination with the original image frame, feature points in the remaining static region in the image frame are identified, and optical flow tracking is carried out; and according to the tracked feature points, calculation of positioning and camera attitude estimation is carried out in combination with IMU pre-integration. According to the method, the target detection technology and the MobileSAM technology are combined to remove high-possibility dynamic objects, so that a universal model can be used to work in different scenes, and the training cost when the system is used in different environments is saved while the accuracy and robustness of the system are kept.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of positioning technology, and in particular relates to a visual inertial odometer in a dynamic environment and a design method and system thereof. Background Art

[0002] With the development of science and technology, high-precision positioning and navigation technology plays an increasingly important role in many applications involving human environments. Technologies such as autonomous driving rely on the development of high-precision positioning and navigation technology. Based on this demand, the localization and mapping (SLAM) technology came into being. The SLAM algorithm has a good performance in a static environment, but in a dynamic environment, the feature points on the dynamic object no longer satisfy the relationship of epipolar geometry. The system itself cannot determine whether the movement of these points is caused by the movement of the object itself or the movement of points in the environment, which ultimately leads to a decrease in positioning accuracy and a serious deviation between the relative positioning trajectory and the actual trajectory. This also affects the reliability and stability of the SLAM algorithm in actual use.

[0003] The most important part of the design of a visual SLAM system in a dynamic environment is to design an algorithm that can determine whether the object in the image frame is dynamic or static while the camera is moving, and process it. The current mainstream method is to use deep learning methods to eliminate the influence of dynamic objects in the environment. However, these methods require training of corresponding models for different dynamic objects in different environments, and their versatility is poor. Therefore, the development of a universal and effective visual inertial odometer positioning technology has great application value and significance. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a visual inertial odometer and a design method and system thereof in a dynamic environment. By combining target detection technology and MobileSAM technology to remove high-probability dynamic objects, a universal model can be used to work in different scenarios. While maintaining the accuracy and robustness of the system, the training cost when used in different environments is saved.

[0005] According to one aspect of the present invention, a design method of a visual inertial odometer in a dynamic environment is provided, comprising: Acquire visual information about the environment; Recognize the visual information and transmit the recognition result as an interactive prompt word; According to the interactive prompt words, the Mobile SAM model is used to perform semantic segmentation of a specific area to obtain a potential high-likelihood dynamic area in the current image frame; According to the potential high-likelihood dynamic area, feature point extraction and optical flow tracking are performed in combination with the original image frame; Based on the tracked feature points, the positioning and camera attitude estimation are calculated in combination with IMU pre-integration.

[0006] As a further technical solution, the visual information is recognized and the recognition result is transmitted as an interactive prompt word, including: Use the target detection model to perform high-probability dynamic target detection; Use the Hungarian algorithm to compensate the accuracy of target detection results; The precision-compensated result is passed as an interactive prompt.

[0007] As a further technical solution, the Hungarian algorithm is used to compensate the accuracy of the target detection results, including: based on the current frame, using the Hungarian algorithm to match the recognition box in the next frame, and using Kalman filtering for prediction; if the detected target suddenly disappears, the Hungarian matching algorithm is used again to compensate for the detected target.

[0008] As a further technical solution, according to the interactive prompt words, the Mobile SAM model is used to perform semantic segmentation of a specific area, including: The coordinates of the recognition frame are used as foreground prompt points of the MobileSAM model, and a number of unrecognized points in the static background are randomly selected as background prompt points of the MobileSAM model; The obtained foreground cue point coordinates and background cue point coordinates are passed into the MobileSAM model for segmentation and image mask generation. The area covered by the mask is a potential high-probability dynamic area.

[0009] As a further technical solution, feature point extraction and tracking are performed based on the potential high-likelihood dynamic area in combination with the original image frame, including: The mask corresponding to the potential high-likelihood dynamic area in the image frame is superimposed with the original image according to the timestamp; Perform feature extraction in the area not covered by the mask; Perform optical flow matching between the previous and next two frames of images; According to the matching results, combined with the mask corresponding to the potential high-probability dynamic area in the image frame, the points in the dynamic area matched by the optical flow in the next frame of the image are deleted.

[0010] As a further technical solution, the method further includes: if there are too few feature points in the static area, compensating the feature points.

[0011] According to one aspect of the present invention, a visual inertial odometer in a dynamic environment is provided, which is designed by using the design method of the visual inertial odometer in a dynamic environment.

[0012] According to one aspect of the present invention, a design system for a visual inertial odometer in a dynamic environment is provided, comprising: The acquisition module is used to obtain visual information in the environment; A recognition module, used for recognizing the visual information and transmitting the recognition result as an interactive prompt word; A semantic segmentation module, used to perform semantic segmentation of a specific area using a Mobile SAM model according to the interactive prompt words, to obtain a potential high-likelihood dynamic area in the current image frame; A feature point extraction and tracking module, used to extract feature points and track optical flow in combination with the original image frame according to the potential high-likelihood dynamic area; The pose calculation module is used to calculate positioning and camera pose estimation based on the tracked feature points combined with IMU pre-integration.

[0013] According to one aspect of the present specification, there is provided an electronic device, comprising: at least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the described method.

[0014] According to one aspect of the present specification, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the method described above.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1) Aiming at the problem that dynamic objects affect positioning accuracy, this paper proposes a design method for visual inertial odometer in dynamic environment. This method uses YOLO target detection technology and novel MobileSAM technology to construct a modular dynamic environment visual SLAM system with strong generalization ability and scalability.

[0016] 2) The present invention uses an improved SORT algorithm to optimize the target detection module. Through the pairing process of Kalman filter prediction and the Hungarian matching algorithm on both sides, the problem of the target detection model suddenly losing the detection target in some frames of images due to occlusion or motion blur when performing multi-target detection and tracking is solved.

[0017] 3) The present invention proposes to use semantic segmentation mask to extract static feature points, ensuring that there are still enough static feature points for subsequent processing after removing the dynamic area, thereby improving the accuracy and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a method for designing a visual inertial odometer in a dynamic environment provided by an embodiment of the present invention.

[0020] Figure 2 A schematic diagram of a technical route for a design method of a visual inertial odometer in a dynamic environment provided by an embodiment of the present invention.

[0021] Figure 3 A schematic diagram of a design system for a visual inertial odometer in a dynamic environment provided by an embodiment of the present invention.

[0022] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] It should be noted that: The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0024] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely exemplary and do not necessarily include all the contents and operations / steps, nor do they necessarily have to be executed in the order described. For example, some operations / steps may be further decomposed, while some operations / steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0026] like Figure 1 As shown, an embodiment of the present invention provides a design method for a visual inertial odometer in a dynamic environment, including: acquiring visual information in the environment; recognizing the visual information and transmitting the recognition result as an interactive prompt word; according to the interactive prompt word, using the Mobile SAM model to perform semantic segmentation of a specific area to obtain a potential high-probability dynamic area in the current image frame; according to the potential high-probability dynamic area, combining the original image frame to perform feature point extraction and optical flow tracking; according to the tracked feature points, combining IMU pre-integration to perform positioning and camera posture estimation calculations.

[0027] The embodiment of the present invention effectively combines the target detection technology with the novel MobileSAM technology to construct a modular dynamic environment visual SLAM system, which has high accuracy and strong generalization ability and scalability.

[0028] like Figure 2 As shown, the technical route of a design method of a visual inertial odometer in a dynamic environment provided by an embodiment of the present invention is as follows: First, the camera captures the information in the environment and passes it to the target recognition module, which recognizes the single frame image and gives the recognition frame of the dynamic object; After that, the SORT algorithm (Hungarian algorithm) is used to match the recognition box in the next frame, and the Kalman filter is used for prediction. When the detection target suddenly disappears, the Hungarian matching algorithm is used again to compensate for the detection target; Next, the Mobile SAM (Segment Anything Model) model is used to segment the image in a specific area. The segmented area is the potential high-probability dynamic area in the current image frame. Next, the result obtained after semantic segmentation by Mobile SAM is input into the feature point extraction and tracking thread together with the original image frame, and the feature points in the remaining static area of ​​the image frame are identified and optical flow tracking is performed. In addition, feature points are compensated when there are too few feature points in the static area. Finally, the attitude estimation thread combines the IMU pre-integration and the feature points tracked in the visual information to perform positioning and camera attitude estimation calculations.

[0029] Specifically, the method comprises the following steps: Step 1: The RGB camera captures the information in the environment and passes it to the target recognition module. The target recognition module uses the yolov8 algorithm to identify the single frame image and give an identification box based on the potential high-probability dynamic objects predefined in the model.

[0030] Step 2: Based on the current frame, use the SORT algorithm to match the recognition box in the next frame and use the Kalman filter for prediction. If the detected target suddenly disappears, use the Hungarian matching algorithm again to compensate for the detected target to ensure continuous and stable tracking of the target in the visual image.

[0031] Step 3: Use the four vertex coordinates of the target recognition box as interactive prompts of the MobileSAM model, and perform image segmentation of a specific area according to the prompts. The segmented area is the potential high-likelihood dynamic area in the current image frame.

[0032] Step 4: Input the result obtained after semantic segmentation of the MobileSAM model together with the original image frame into the feature point extraction and tracking thread, identify the feature points in the remaining static area of ​​the image frame and perform optical flow tracking, and compensate for the feature points when there are too few feature points in the static area.

[0033] Step 5: The attitude estimation thread combines the IMU pre-integration with the feature points tracked in the visual information to perform positioning and camera attitude estimation calculations.

[0034] Furthermore, step 1 is specifically as follows: the picture captured by the camera is transmitted to yolov8 for target detection processing, the pre-defined high-probability moving targets in the current frame are identified, and the center coordinates of all identified targets in the frame image are recorded for subsequent use.

[0035] Furthermore, step 2 is specifically as follows: establishing a state model describing the target and using the Hungarian matching algorithm for association detection, and then using Kalman filtering to update the target state. The target state model used in the embodiment of the present invention represents the state of the target as a vector: (1) Where x, y represent the center position of the target, w, h represent the length and width of the target detection box, Indicates the moving speed of the target center position, represents the time rate of change of the length and width of the target detection box, and S represents the area of ​​the target detection box. In addition, we use the intersection-over-union (IOU) between target detection boxes to generate a cost matrix and use the Hungarian algorithm to optimally solve the allocation. The optimal solution obtained by the Hungarian matching algorithm is used as the basis for subsequent Kalman filtering to ensure that each state variable corresponds to the corresponding target. After the target is detected and matched, the detected bounding box is used to update the target state, and its state variables are optimally solved through the Kalman filter framework. If the target is not detected, the new target state is predicted based on the previous target state through the linear Kalman filter framework and compensated to the current image.

[0036] According to the linear relationship between these variables, the following set of state equations can be obtained: (2) According to the above linearly constructed state variables, the state transfer matrix F can be constructed (3) Define H as the measurement matrix, which is the prior estimate vector of the current stage prediction output Each state value is converted into a sensor column vector The mapping relationship between the corresponding measurement values.

[0037] (4) definition is the sensor covariance noise matrix, K is the Kalman gain, then the Kalman gain satisfies (5) From the above, we can get the state update equation of the whole process as follows: (6) in, It refers to the prior state estimate at time k, that is, the result at time k predicted based on the optimal estimate at the previous time (time k-1). is the a posteriori state estimate at time k, which is the update of the state after receiving the measurement information at time k. Represents the prior estimated covariance matrix of the state variable at time k-1 for the prediction of time k, that is, The covariance matrix of . Represents the posterior estimated covariance matrix of the state variables, that is The covariance matrix of .

[0038] Furthermore, step 3 is specifically as follows: passing the original image frame and the result after target detection accuracy compensation to semantic segmentation, using the obtained coordinates of the target box as the foreground cue point of the MobileSAM model, randomly selecting a number of unrecognized points in the static background as the background cue points of the MobileSAM model, and passing the obtained foreground cue point coordinates and background cue point coordinates into MobileSAM for segmentation and generating an image mask. The area covered by the mask is the area where the dynamic target exists. The area covered by the mask is eliminated to obtain the static part of the environment for subsequent feature point extraction and positioning operations.

[0039] Furthermore, step 4 is specifically as follows: in the feature point extraction and tracking thread, the feature points in the remaining static area of ​​the image frame after step 3 are identified and optical flow tracking is performed, the tracked feature points are sorted according to the number of times they are tracked and points are selected in sequence, and dense points are removed to make the feature points evenly distributed. If there are too few feature points in the static area, feature point compensation is performed.

[0040] As a preferred embodiment, the feature point compensation method provided by the embodiment of the present invention is as follows: Assume that the maximum number of feature points detected in the i-1th frame is , the feature point set is (7) If the optical flow matching is performed between the i-1th frame and the i-th frame, The points that can be matched in the i-th frame image are , since the relative position of the object between the two frames has changed, it is necessary to determine whether these s points are in the mask coverage area of ​​the i-th frame image. , determine whether the corresponding value of each point in the mask MASK is 0, and exclude the points whose corresponding value is 0. Assume that this process excludes Points, remaining Points are recorded , then a new round of static feature point extraction should be performed in the i-th frame. In this round of feature point extraction, the maximum number of feature point extraction is set to , assuming that the feature points extracted from this frame are , since the number of extracted feature points is generally less than the set maximum value, .Will As compensation and Merge as the next frame , (8) This can ensure that the visual SLAM has enough reliable static feature points to perform pose calculations, and can ensure the stable operation of the system even when the dynamic area in the image is large, thereby improving the robustness of the visual SLAM system in complex dynamic environments.

[0041] Furthermore, step 5 is specifically as follows: using the attitude estimation thread in the VINS-Mono system, and combining the IMU pre-integration with the feature points tracked in the visual information to perform positioning and camera attitude estimation calculations, and finally obtaining the positioning result.

[0042] The embodiment of the present invention mainly uses the YOLO target detection technology combined with the MobileSAM technology to remove high-probability dynamic objects. It can work in different scenarios using a universal model, while maintaining the accuracy and robustness of the system, saving the training cost when used in different environments.

[0043] An embodiment of the present invention further provides a visual inertial odometer in a dynamic environment, which is designed using the design method of the visual inertial odometer in a dynamic environment described in the above embodiment.

[0044] The implementation basis of each embodiment of the present invention is realized by programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a design system for a visual inertial odometer in a dynamic environment, and the device is used to execute the design system method of a visual inertial odometer in a dynamic environment in the above method embodiment.

[0045] See also Figure 3 The system includes: an acquisition module for acquiring visual information in the environment; a recognition module for recognizing the visual information and transmitting the recognition result as an interactive prompt word; a semantic segmentation module for performing semantic segmentation of a specific area using a Mobile SAM model according to the interactive prompt word to obtain a potential high-probability dynamic area in the current image frame; a feature point extraction and tracking module for performing feature point extraction and optical flow tracking based on the potential high-probability dynamic area in combination with the original image frame; and a posture calculation module for performing positioning and camera posture estimation calculation based on the tracked feature points in combination with IMU pre-integration.

[0046] The embodiment of the present invention provides a design system for a visual inertial odometer in a dynamic environment. The design of the existing visual SLAM system in a dynamic environment requires training of corresponding models for different dynamic objects in different environments, which has poor versatility. Figure 3Several modules in the system are combined with target detection technology and MobileSAM technology to remove dynamic objects with high probability, so that a universal model can be used to work in different scenarios. While maintaining the accuracy and robustness of the system, it saves the training cost when used in different environments.

[0047] It should be noted that the system embodiment provided by the present invention is used to implement the methods in the above method embodiment as well as the methods in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set. The principle is basically the same as the principle of the above system embodiment provided by the present invention. As long as the technical personnel in this field refer to the specific technical solutions in other method embodiments on the basis of the above system embodiment, obtain the corresponding technical means and the technical solutions composed of these technical means by combining the technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above system embodiment to obtain the corresponding system class embodiments, which are used to implement the methods in other method class embodiments. For example: Based on the contents of the above system embodiments, as a preferred embodiment, a design system for a visual inertial odometer in a dynamic environment provided in an embodiment of the present invention further includes: A first recognition submodule, for performing high-probability dynamic target detection using a target detection model; The second recognition submodule is used to use the Hungarian algorithm to perform accuracy compensation on the target detection result; The third recognition submodule is used to transmit the result after accuracy compensation as an interactive prompt word.

[0048] Based on the content of the above system embodiment, as a preferred embodiment, a design system of a visual inertial odometer in a dynamic environment provided in an embodiment of the present invention, the second identification submodule shown also includes: Based on the current frame, the Hungarian algorithm is used to match the recognition box in the next frame, and the Kalman filter is used for prediction; If the detected target disappears suddenly, the Hungarian matching algorithm is used again to compensate for the detected target.

[0049] Based on the contents of the above system embodiments, as a preferred embodiment, a design system for a visual inertial odometer in a dynamic environment provided in an embodiment of the present invention further includes: The first semantic segmentation submodule is used to use the coordinates of the recognition box as the foreground prompt point of the MobileSAM model, and randomly select a number of unrecognized points in the static background as the background prompt points of the MobileSAM model; The second semantic segmentation submodule is used to pass the obtained foreground cue point coordinates and background cue point coordinates into the MobileSAM model for segmentation and generate an image mask. The area covered by the mask is a potential high-probability dynamic area.

[0050] Based on the contents of the above system embodiments, as a preferred embodiment, a design system for a visual inertial odometer in a dynamic environment provided in an embodiment of the present invention further includes: The first feature point extraction and tracking submodule is used to superimpose the mask corresponding to the potential high-likelihood dynamic area in the image frame with the original image according to the timestamp; The second feature point extraction and tracking submodule is used to extract features in the area not covered by the mask; The third feature point extraction and tracking submodule is used to perform optical flow matching between the previous and next two frames of images; The fourth feature point extraction and tracking submodule is used to delete the points in the dynamic area matched by the optical flow in the next frame of image according to the matching results and the mask corresponding to the potential high-probability dynamic area in the image frame.

[0051] Based on the contents of the above system embodiments, as a preferred embodiment, a design system of a visual inertial odometer in a dynamic environment provided in an embodiment of the present invention further includes: if there are too few feature points in the static area, compensating the feature points.

[0052] Based on the contents of the above system embodiments, as a preferred embodiment, a design system for a visual inertial odometer in a dynamic environment provided in an embodiment of the present invention further includes: The pose calculation submodule is used to use the pose estimation thread in the VINS-Mono system and combine the IMU pre-integration with the feature points tracked in the visual information to perform positioning and camera pose estimation calculations to ultimately obtain the positioning result.

[0053] The method of the embodiment of the present invention is implemented by relying on electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 4 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein at least one processor, the communication interface, and at least one memory communicate with each other through the communication bus. At least one processor calls the logic instructions in at least one memory to execute all or part of the steps of the method provided by the aforementioned various method embodiments.

[0054] In addition, when the logic instructions in the above-mentioned at least one memory are implemented in the form of a software functional unit and sold or used as an independent product, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or partly in other words, the part that contributes to the prior art or part of the technical solution is embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for enabling a computer device (a personal computer, a server, or a network device) to perform all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and various media for storing program codes.

[0055] The system embodiments described above are merely illustrative, wherein the units described as separate components are or are not physically separated, and the components shown as units are or are not physical units, located in one place, or distributed on multiple network units. Some or all of the modules are selected according to actual conditions to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement without creative effort.

[0056] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0058] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0060] In summary, the present invention aims at the problem that the existing methods need to train corresponding models for different dynamic objects in different environments and have poor versatility, and provides a design method for a visual inertial odometer in a dynamic environment, including: first, the camera captures the information in the environment and transmits it to the target recognition module, and the target recognition module recognizes the single frame image according to the potential high-likelihood dynamic objects predefined in the model and gives a recognition frame. Secondly, the recognition frame in the next frame is matched using the Hungarian matching algorithm using the SORT algorithm, and the Kalman filter is used for prediction. When the detection target suddenly disappears, the Hungarian matching algorithm is used again to compensate the detection target. Then, the four vertex coordinates of the target recognition frame are used as the interactive prompt words of the Mobile SAM model to perform image segmentation of a specific area, and the segmented area is the potential high-likelihood dynamic area in the current image frame. Next, the result obtained after the semantic segmentation of the MobileSAM model is input into the feature point tracking and compensation thread together with the original image frame, the feature points in the remaining static area in the image frame are identified and optical flow tracking is performed, and the feature points are compensated when there are too few feature points in the static area. Finally, the attitude estimation thread combines the IMU pre-integration with the feature points tracked in the visual information to perform positioning and camera attitude estimation calculations.

[0061] The present invention effectively combines target detection technology with the novel MobileSAM technology to construct a modular dynamic environment visual SLAM system, which has high accuracy and strong generalization ability and scalability.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A design method for a visual inertial odometer in a dynamic environment, characterized in that: include: Acquire visual information about the environment; Recognize the visual information and transmit the recognition result as an interactive prompt word; According to the interactive prompt words, the Mobile SAM model is used to perform semantic segmentation of a specific area to obtain a potential high-likelihood dynamic area in the current image frame; According to the potential high-likelihood dynamic area, feature point extraction and optical flow tracking are performed in combination with the original image frame; Based on the tracked feature points, the positioning and camera attitude estimation are calculated in combination with IMU pre-integration.

2. According to the design method of a visual inertial odometer in a dynamic environment as described in claim 1, it is characterized in that: Recognizing the visual information and delivering the recognition result as an interactive prompt word, including: Use the target detection model to perform high-probability dynamic target detection; Use the Hungarian algorithm to compensate the accuracy of target detection results; The precision-compensated result is passed as an interactive prompt.

3. According to the design method of a visual inertial odometer in a dynamic environment as described in claim 2, it is characterized in that: The Hungarian algorithm is used to compensate the accuracy of the target detection results, including: based on the current frame, the Hungarian algorithm is used to match the recognition box in the next frame, and the Kalman filter is used for prediction; if the detected target suddenly disappears, the Hungarian matching algorithm is used again to compensate for the detected target.

4. According to the design method of a visual inertial odometer in a dynamic environment as described in claim 1, it is characterized in that: According to the interactive prompt words, the Mobile SAM model is used to perform semantic segmentation of a specific area, including: The coordinates of the recognition frame are used as foreground prompt points of the MobileSAM model, and a number of unrecognized points in the static background are randomly selected as background prompt points of the MobileSAM model; The obtained foreground cue point coordinates and background cue point coordinates are passed into the MobileSAM model for segmentation and image mask generation. The area covered by the mask is a potential high-probability dynamic area.

5. According to the design method of a visual inertial odometer in a dynamic environment as described in claim 4, it is characterized in that: According to the potential high-likelihood dynamic area, feature point extraction and tracking are performed in combination with the original image frame, including: The mask corresponding to the potential high-likelihood dynamic area in the image frame is superimposed with the original image according to the timestamp; Perform feature extraction in the area not covered by the mask; Perform optical flow matching between the previous and next two frames of images; According to the matching results, combined with the mask corresponding to the potential high-probability dynamic area in the image frame, the points in the dynamic area matched by the optical flow in the next frame of the image are deleted.

6. The design method of a visual inertial odometer in a dynamic environment according to claim 5, characterized in that: The method further includes: if there are too few feature points in the static area, compensating the feature points.

7. A visual inertial odometer in a dynamic environment, characterized in that: The method for designing a visual inertial odometer in a dynamic environment according to any one of claims 1 to 6 is adopted to design the device.

8. A design system for visual inertial odometer in dynamic environment, characterized in that: include: The acquisition module is used to obtain visual information in the environment; A recognition module, used for recognizing the visual information and transmitting the recognition result as an interactive prompt word; A semantic segmentation module, used to perform semantic segmentation of a specific area using a Mobile SAM model according to the interactive prompt words, to obtain a potential high-likelihood dynamic area in the current image frame; A feature point extraction and tracking module, used to extract feature points and track optical flow in combination with the original image frame according to the potential high-likelihood dynamic area; The pose calculation module is used to calculate positioning and camera pose estimation based on the tracked feature points combined with IMU pre-integration.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; The memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 6.

10. A non-transitory computer readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the method of any one of claims 1 to 6.

Citation Information

Cited By

  • Anti-jitter method and device based on vehicle-mounted pavement disease recognition system

    CN119228849A

  • Path planning method and system based on multi-modal joint integration

    CN119642805A

  • Unmanned aerial vehicle lightweight semantic visual inertial odometer method, medium and processing equipment

    CN119879857A

  • Unmanned aerial vehicle lightweight semantic visual inertial odometry method, medium and processing device

    CN119879857B