Visual-inertial slam method and system suitable for complex dynamic environments

Abstract

The application discloses a visual-inertial SLAM method, system, storage medium and electronic equipment suitable for a complex dynamic environment. The method comprises the following steps: acquiring a visual image sequence; the visual image sequence comprises a plurality of visual images; using a time communication mechanism to synchronize instance segmentation of the visual images, extract key frames, and perform feature extraction and matching on the key frames to obtain a feature matching pair; using a semantic-assisted RANSAC method to remove static outliers from the feature matching pair, and identifying dynamic feature points in combination with instance information and Sampson geometric information; constructing an IMU pre-integration residual error and an edge residual error, removing the identified dynamic feature points, constructing a visual re-projection residual error using the remaining static feature points, and calculating an optimized pose through a solver. The application can improve the accuracy and stability of dynamic feature identification in a complex dynamic scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a visual-inertial SLAM method and system suitable for complex dynamic environments, a storage medium and an electronic device. BACKGROUND

[0002] With the continuous progress of computer technology, sensor technology, artificial intelligence, Internet of Things and other fields, mobile robots can perform autonomous navigation, decision-making and operation in complex environments, thereby reducing the workload of humans, improving work efficiency and safety, and being widely applied in manufacturing, medical treatment, services, security, transportation and other fields, with a growing market size, becoming an important force to promote social progress and development.

[0003] Real-time positioning and mapping (Simultaneous Localization and Mapping, SLAM) technology can help robots to realize autonomous positioning and environment reconstruction in unknown environments using various sensors, and is a key technology for mobile robots to realize autonomous navigation, environment exploration and other tasks. According to the use category of sensors, SLAM is mainly divided into laser SLAM, visual SLAM and multi-source SLAM. Laser SLAM uses a laser radar to obtain point cloud information in the environment, realizes high-precision positioning and mapping, but has high cost. Visual SLAM uses a camera sensor to obtain image features and uses computer vision technology to complete positioning and pose determination, has low cost, and is widely used in robot systems. However, monocular visual SLAM has scale uncertainty and is difficult to accurately recover the pose. An inertial measurement unit (Inertial Measurement Unit, IMU) can obtain high-frequency inertial information of the carrier without being affected by the external environment, and can be used to assist visual SLAM. Therefore, a visual-inertial SLAM system based on a visual camera and an IMU has become one of the core technologies for mobile robots to realize autonomous navigation in complex environments. However, existing visual-inertial SLAM systems are mostly suitable for static environments.

[0004] In a static scene, the feature points in the environment remain stationary, satisfying the epipolar geometry constraint relationship in computer vision. The visual camera carried on the mobile robot uses these static features to construct visual re-projection constraint information, and realizes accurate estimation of the pose through nonlinear optimization or filtering. However, in common complex urban scenes, there are a large number of moving objects, such as moving cars and walking pedestrians. The feature points extracted from these dynamic objects no longer satisfy the epipolar geometry constraint relationship. These dynamic features will introduce additional noise and uncertainty, seriously affecting the accuracy and stability of the back-end optimization, causing the mobile robot pose estimation to deviate, and making it difficult to maintain the consistency of the environment map. Therefore, the visual-inertial SLAM system for complex dynamic environments is difficult to realize high-precision pose estimation.

[0005] For dynamic scenes, current SLAM techniques such as VINS-Mono use robust kernel functions to mitigate the negative impact of dynamic feature outliers on optimization, but this method can only reject slowly moving features and is ineffective for fast moving features. Some SLAM systems combine artificial intelligence techniques and use deep learning methods to obtain high-level instance information in images, and combine traditional geometric information to identify dynamic features, which have strong adaptability in dynamic environments, but these methods do not consider system running efficiency when introducing deep learning techniques, which seriously affects the real-time performance of the system. At the same time, the combination of geometric information and instance information is relatively simple, and it is difficult to guarantee the accuracy and stability of dynamic feature recognition.

[0006] In summary, the presence of moving objects in complex dynamic scenes severely affects the positioning accuracy and robustness of visual-inertial SLAM systems. The traditional geometric method in existing dynamic SLAM methods is easily affected by observation noise, while the method that introduces deep learning technology has the problem of simple information utilization and low system running efficiency. SUMMARY

[0007] The embodiments of the present application provide a visual-inertial SLAM method, system, storage medium and electronic device suitable for complex dynamic environments, which can improve the accuracy and stability of dynamic feature recognition in complex dynamic scenes.

[0008] The embodiments of the present application provide a visual-inertial SLAM method suitable for complex dynamic environments, comprising:

[0009] obtaining a sequence of visual images; the sequence of visual images includes a plurality of visual images;

[0010] synchronously performing instance segmentation on the visual images to extract key frames and performing feature extraction and matching on the key frames using a time communication mechanism to obtain a feature matching pair;

[0011] using a semantic-assisted RANSAC method to remove static outliers from the feature matching pair, and combining instance information and Sampson geometric information to identify dynamic feature points;

[0012] constructing IMU pre-integrated residuals and marginalized residuals to remove the identified dynamic feature points, constructing visual re-projection residuals using the remaining static feature points, and calculating the optimized pose through a solver.

[0013] Further, the above-mentioned visual-inertial SLAM method suitable for complex dynamic environments, wherein the synchronously performing instance segmentation on the visual images to extract key frames and performing feature extraction and matching on the key frames using a time communication mechanism to obtain a feature matching pair comprises:

[0014] In the neural network thread, time information corresponding to a received visual image is transmitted to a VIO thread, instance segmentation is performed on the visual image to obtain instance pixels and instance IDs, an instance mask is constructed based on the instance pixels and the instance IDs, and the instance mask is transmitted to the VIO thread.

[0015] In the VIO thread, according to the time information, a corresponding visual image is extracted from the image sequence, feature tracking is performed on feature points at a previous time by an optical flow method to obtain a feature matching pair, tracking points that fail to track and exceed a boundary are eliminated, and new corner points are extracted as supplementary feature points.

[0016] Gray pixels at corresponding positions in the instance mask are taken as instance ID information, and the instance ID information is associated and matched with corresponding feature points.

[0017] Further, the above visual-inertial SLAM method suitable for a complex dynamic environment, wherein the constructing of the instance mask based on the instance pixels and the instance IDs comprises:

[0018] Constructing a gray image based on the instance pixels and corresponding instance IDs;

[0019] For different instance pixel overlapping regions, instance ID coverage is performed on the gray image according to a segmentation order;

[0020] The covered gray image is taken as the instance mask.

[0021] Further, the above visual-inertial SLAM method suitable for a complex dynamic environment, wherein the using of the RANSAC method assisted by semantics to eliminate static outliers of the feature matching pair comprises:

[0022] The feature matching pair is taken as a prior point set, and feature points belonging to potential dynamic semantics of the feature matching pair are eliminated;

[0023] According to instance ID information to which each feature belongs, the feature matching pair after elimination of the feature points belonging to potential dynamic semantics is divided into different clusters;

[0024] A fundamental matrix of adjacent key frames is calculated, outliers in the feature matching pair are eliminated based on the fundamental matrix, and the feature points belonging to potential dynamic semantics that are eliminated before are added.

[0025] Further, the above visual-inertial SLAM method suitable for a complex dynamic environment, wherein dynamic feature points are identified in combination with instance information and Sampson geometric information, comprising:

[0026] calculating epipolar error information of each instance object based on the fundamental matrix, and adding the epipolar error information to a geometric error sequence of the instance object;

[0027] For each instance object, a proportion of points in the geometric error sequence that are greater than a preset error threshold is counted, and if the proportion is greater than 80%, the instance object is determined to be in a motion state, and all feature points belonging to the instance object are regarded as dynamic features.

[0028] Further, the above-mentioned visual-inertial SLAM method suitable for a complex dynamic environment, wherein the formula for calculating the epipolar error information is:

[0029]

[0030] wherein, , are homogeneous coordinates of pixel coordinates in adjacent key frames and respectively, is a fundamental matrix, and T represents transposition.

[0031] Further, the above-mentioned visual-inertial SLAM method suitable for a complex dynamic environment, wherein the constructing of the IMU pre-integration residual and the marginalization residual, the elimination of the identified dynamic feature points, the construction of the visual re-projection residual using the remaining static feature points, and the calculation of the optimized pose by the solver include:

[0032] calculating the IMU pre-integration residual of the adjacent key frames;

[0033] calculating the marginalization residual according to the marginalization information of the sliding window;

[0034] traversing all feature points, eliminating the identified dynamic features, and constructing the visual re-projection residual of the starting observation frame and other observation frames using the remaining static features;

[0035] solving the residual function in the solver using the DogLeg method to obtain the optimized pose.

[0036] Embodiments of the present application also provide a visual-inertial SLAM system suitable for a complex dynamic environment, comprising:

[0037] an acquisition module configured to acquire a visual image sequence; the visual image sequence comprises a plurality of visual images;

[0038] a key frame elastic regulation module configured to extract key frames by instance segmentation and extract and match features of the key frames using a time communication mechanism to synchronize the visual images, to obtain a feature matching pair;

[0039] A cascaded visual outlier rejection module is configured to perform static outlier rejection on the feature matching pairs using a semantic-aided RANSAC method, and identify dynamic feature points in combination with instance information and Sampson geometric information;

[0040] A backend optimization module is configured to construct IMU pre-integrated residuals and marginalized residuals, reject the identified dynamic feature points, construct visual re-projection residuals using remaining static feature points, and calculate an optimized pose by a solver.

[0041] The embodiments of the present application further provide a computer readable storage medium, which stores a plurality of instructions. The instructions are adapted to be loaded by a processor to execute any of the above-mentioned visual-inertial SLAM methods suitable for complex dynamic environments.

[0042] The embodiments of the present application further provide an electronic device, which comprises a processor and a memory. The processor is electrically connected to the memory. The memory is configured to store instructions and data. The processor is configured to perform the steps in any of the above-mentioned visual-inertial SLAM methods suitable for complex dynamic environments.

[0043] The visual-inertial SLAM method, system, storage medium and electronic device provided by the present application can flexibly adapt to different neural networks, elastically regulate key frames, process multi-level visual observation information, and improve system running efficiency and stability. In addition, the present application further proposes a semantic-aided RANSAC algorithm on the basis of the RANSAC algorithm, which effectively reduces the proportion of outliers in the initial point set and improves the static outlier rejection accuracy. At the same time, the dynamic features are identified based on the characteristics of instance feature motion consistency and Sampson geometric error information, which improves the dynamic outlier rejection accuracy. The positioning accuracy and robustness of the visual-inertial SLAM are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] The technical solutions and other beneficial effects of the present application will become apparent through the following detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings.

[0045] Figure 1 A flowchart of the visual-inertial SLAM method suitable for complex dynamic environments provided by the embodiments of the present application is shown.

[0046] Figure 2 Another flowchart of the visual-inertial SLAM method suitable for complex dynamic environments provided by the embodiments of the present application is shown.

[0047] Figure 3A flowchart of synchronously performing instance segmentation and feature extraction is provided for the embodiments of the present application.

[0048] Figure 4 A flowchart of calculating an optimized pose is provided for the embodiments of the present application.

[0049] Figure 5 A structural schematic diagram of a visual-inertial SLAM system suitable for a complex dynamic environment is provided for the embodiments of the present application.

[0050] Figure 6 A structural schematic diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0052] The embodiments of the present application provide a visual-inertial SLAM method, system, storage medium and electronic device suitable for a complex dynamic environment. The visual-inertial SLAM system suitable for a complex dynamic environment provided by the embodiments of the present application can be integrated in an electronic device, which can be a terminal, a server or the like, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box or other devices, etc.

[0053] Please refer to Figure 1 With Figure 2 , Figure 1 A flowchart of a visual-inertial SLAM method suitable for a complex dynamic environment is provided for the embodiments of the present application, Figure 2 Another flowchart of a visual-inertial SLAM method suitable for a complex dynamic environment is provided for the embodiments of the present application, which is applied in an electronic device, and the visual-inertial SLAM method suitable for a complex dynamic environment includes the following steps:

[0054] S1, acquiring a visual image sequence; the visual image sequence includes a plurality of visual images.

[0055] The visual image sequence is acquired by a camera, and the visual image sequence includes a plurality of visual images and time information corresponding to each image.

[0056] S2, synchronously performing instance segmentation on the visual images by using a time communication mechanism to extract key frames and performing feature extraction and matching on the key frames to obtain a feature matching pair.

[0057] Specifically, the instance segmentation key frame and the key frame geometric feature extraction are synchronized by using a time communication mechanism to flexibly control the interval between key frames. Figure 3 A flowchart for synchronously performing instance segmentation and feature extraction provided by an embodiment of the present application is shown in Figure 3 The time communication mechanism synchronization is achieved through a Visual Inertial Odometry (VIO) thread.

[0058] In an embodiment, step S2 includes the following steps:

[0059] S21, in the neural network thread, transmitting time information corresponding to the received visual image to the VIO thread, performing instance segmentation on the visual image to obtain instance pixels and instance IDs, constructing an instance mask based on the instance pixels and the instance IDs, and transmitting the instance mask to the VIO thread.

[0060] Specifically, the visual image can be segmented by a MaskRCNN network.

[0061] In an embodiment, the instance segmentation on the visual image in step S21 to obtain instance pixels and instance IDs specifically includes:

[0062] S211, constructing a grayscale image based on the instance pixels and the corresponding instance IDs;

[0063] S212, for different instance pixel overlapping regions, performing instance ID covering on the grayscale image according to the segmentation order;

[0064] S213, taking the covered grayscale image as an instance mask.

[0065] S22, in the VIO thread, extracting corresponding visual images in the image sequence according to the time information, performing feature tracking on the feature points at the previous time by an optical flow method to obtain a feature matching pair, eliminating tracking points that fail to track and exceed the boundary, and extracting new Shi-Tomasi corner points as supplementary feature points to meet the requirement for the number of feature points.

[0066] S23, taking the grayscale pixels at the corresponding positions in the instance mask as instance ID information, and associating and matching the instance ID information with the corresponding feature points.

[0067] S3, using a semantic auxiliary RANSAC method to eliminate static outliers from the feature matching pair, and identifying dynamic feature points.

[0068] In an embodiment, the semantic auxiliary RANSAC method used in step S3 to eliminate static outliers from the feature matching pair includes the following steps:

[0069] S31, remove feature points belonging to potential dynamic semantics from the feature matching pairs as the prior point set.

[0070] Specifically, remove feature points belonging to potential dynamic semantics such as vehicles, pedestrians, etc. from the feature matching pairs as the prior point set, so as to reduce the proportion of outliers in the prior point set.

[0071] S32, divide the feature matching pairs after removing the feature points belonging to potential dynamic semantics into different clusters according to the instance ID information to which each feature belongs.

[0072] The motion state of the feature points on the object should be consistent, so the features belonging to different instances are divided into different clusters according to the instance ID information to which each feature in the image belongs.

[0073] S33, calculate the fundamental matrix of the adjacent key frames, remove outliers in the feature matching pairs based on the fundamental matrix, and then add the feature points belonging to potential dynamic semantics removed before.

[0074] Specifically, the fundamental matrix of the relative pose of the camera coordinate system of the adjacent key frames and is calculated , and the calculation method of the fundamental matrix is as follows:

[0075]

[0076] wherein, is the extrinsic rotation matrix of the camera and the IMU, is the camera intrinsic matrix, and the fundamental matrix describes the relative pose transformation relationship of the visual pixels. and are and the corresponding carrier coordinate systems, is the world coordinate system. The carrier pose corresponding to the previous frame is , wherein is the rotation matrix of relative to , is the translation vector of relative to . Similarly, the carrier pose corresponding to the current time is , .

[0077] In an embodiment, the dynamic feature points are identified by combining the instance information and the Sampson geometric information in step S3, including the following steps:

[0078] S34, calculating polar line error information of each instance object based on the fundamental matrix, and adding the polar line error information to the geometric error sequence of the instance object.

[0079] wherein the geometric error is a point error between the feature points, and the point error is obtained by calculating the Euclidean distance between two feature points.

[0080] For the feature points belonging to a certain instance object , the homogeneous coordinates of the pixel coordinates in the adjacent key frames and are and respectively. The calculation formula of the Sampson polar line error information is:

[0081]

[0082] wherein is the fundamental matrix, and T represents transposition.

[0083] S35, for each instance object, the proportion of the points in the geometric error sequence that are greater than the preset error threshold is counted, and if the proportion is greater than 80%, the instance object is determined to be in a motion state, and all the feature points belonging to the instance object are regarded as dynamic features.

[0084] In an embodiment, the Sampson polar line error threshold can be 1.0.

[0085] S4, constructing an IMU pre-integration residual error and an edge residual error, eliminating the identified dynamic feature points, constructing a visual re-projection residual error by using the remaining static feature points, and calculating an optimized pose by a solver.

[0086] In an embodiment, Figure 4 the flowchart for calculating the optimized pose provided by the embodiment of the present application is shown as Figure 4 , and step S4 includes the following steps:

[0087] S41, calculating the IMU pre-integration residual error of the adjacent key frames.

[0088] Specifically, first, the angular velocity and acceleration are obtained from the IMU, and then pre-processing is performed, and at the first image key frame, the velocity, position and rotation are initialized. Then, the median integral of each pair of adjacent IMU data points is calculated, the median integral includes the median acceleration and the median angular velocity, and the velocity, position and rotation are updated by the median acceleration and the median angular velocity. For two key frames, all IMU data points between them are integrated by the median, and the change of relative motion (including relative velocity, relative position and relative rotation) is obtained, which is the IMU pre-integral of adjacent key frames. The IMU pre-integral residual is obtained by calculating the difference between the pre-integral relative rotation and the visual estimation relative rotation, the difference between the pre-integral relative velocity and the visual estimation relative velocity, and the difference between the pre-integral relative position and the visual estimation relative position.

[0089] S42, the marginalization residual is calculated according to the marginalization information of the sliding window.

[0090] Specifically, first, the objective function of the pose optimization problem is constructed, the state variables are divided into marginalized states and remaining states, the marginalized states are removed from the pose optimization problem by Schur complement, and the removed prior error term is calculated.

[0091] S43, all feature points are traversed, the identified dynamic features are removed, and the remaining static features are used to construct the visual re-projection residual of the starting observation frame and other observation frames.

[0092] Specifically, first, the coordinates of the map points projected into the camera coordinate system are calculated, and then the coordinates are projected into the image plane. The coordinates projected into the image plane are subtracted from the coordinates of the feature points observed by each frame of image, and the re-projection error is obtained. For each map point, the re-projection error in all observation frames is recorded to form a sequence of visual re-projection residuals.

[0093] S44, the DogLeg method is used to solve the residual function in the solver to obtain the optimized pose.

[0094] The IMU pre-integral residual, the marginalization residual and the visual re-projection residual are used as the residual function, and the DogLeg method is used to solve the residual function to obtain the optimized pose.

[0095] The present application has the following advantages:

[0096] 1) The present application introduces a key frame elastic regulation architecture to solve the problem of decreased running efficiency after introducing deep learning technology in dynamic SLAM, realizes integrated running of instance segmentation network and VIO double thread, can flexibly adapt to different neural networks, elastically regulates key frames, processes multi-level visual observation information, and improves system running efficiency and stability.

[0097] 2) The application proposes a semantic aided RANSAC algorithm based on the RANSAC algorithm, effectively reduces the proportion of initial point set outliers, and improves the precision of static outlier elimination. At the same time, based on the characteristics of instance feature motion consistency and Sampson geometric error information, dynamic features are identified, and the precision of dynamic outlier elimination is improved. The positioning accuracy and robustness of visual inertial SLAM are significantly improved.

[0098] According to the method described in the above embodiment, the present embodiment will be further described from the perspective of a visual inertial SLAM system suitable for a complex dynamic environment, which can be implemented as an independent entity or integrated in an electronic device, which can be a terminal, a server, etc. The terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, etc.

[0099] Please refer to Figure 5 , Figure 5 The application embodiment provides a visual inertial SLAM system suitable for a complex dynamic environment, which is applied to an electronic device. The visual inertial SLAM system suitable for a complex dynamic environment can include:

[0100] The acquisition module is configured to acquire a visual image sequence. The visual image sequence includes a plurality of visual images.

[0101] The key frame elastic regulation module is configured to use a time communication mechanism to synchronize instance segmentation of the visual images to extract key frames, and to extract and match features of the key frames to obtain a feature matching pair.

[0102] The cascaded visual outlier elimination module is configured to use a semantic aided RANSAC method to eliminate static outliers from the feature matching pair, and to identify dynamic feature points in combination with instance information and Sampson geometric information.

[0103] The backend optimization module is configured to construct an IMU pre-integration residual error and an edge residual error, eliminate the identified dynamic feature points, construct a visual re-projection residual error using the remaining static feature points, and calculate an optimized pose through a solver.

[0104] In specific implementation, each of the above modules and / or units can be implemented as an independent entity, or can be combined as the same or several entities. The specific implementation of each of the above modules and / or units can be referred to the method embodiments above, and the beneficial effects that can be achieved can also be referred to the beneficial effects in the method embodiments above, which will not be described here.

[0105] In addition, the application further provides an electronic device, which can be a computer, a tablet computer, or the like. The electronic device can implement the steps in any embodiment of the visual-inertial SLAM method suitable for a complex dynamic environment provided by the embodiments of the application, and thus can implement the beneficial effects of any visual-inertial SLAM method suitable for a complex dynamic environment provided by the embodiments of the application. For details, refer to the foregoing embodiments, which will not be described here again.

[0106] Figure 6 A specific structural block diagram of an electronic device provided by the embodiments of the application is shown, which can be used to implement the visual-inertial SLAM method suitable for a complex dynamic environment provided in the foregoing embodiments. The electronic device 500 can be a terminal, a server, or the like, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, etc.

[0107] RF circuit 510 is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals and vice versa, thereby enabling communication with communication networks or other devices. RF circuit 510 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, subscriber identity modules (SIM cards), memory, etc. RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.

[0108] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above-described embodiments, and the processor 580 can execute various functions and data processing by running the software programs and modules stored in the memory 520, i.e., realize functions such as front camera shooting, processing of the shot image, and switching of display color of the display content on the display screen. The memory 520 can include a high-speed random access memory, and can further include a non-volatile memory such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 520 can further include memories disposed remotely with respect to the processor 580, which can be connected to the electronic device 500 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0109] The input unit 530 can be used to receive inputted digital or character information, and generate a keyboard, a mouse, and the like related to user settings and function control.

[0110] The display unit 540 can be used to display information inputted by the user or provided to the user, and various graphical user interfaces which can be constituted by graphics, text, icons, video, and any combination thereof. The display unit 540 can include a display panel 541, which can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0111] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can convert received audio data into an electrical signal, transmit the electrical signal to the speaker 561, and convert the electrical signal into a sound signal outputted by the speaker 561; on the other hand, the microphone 562 can convert a sound signal collected into an electrical signal, and the audio circuit 560 can convert the electrical signal into audio data, output the audio data to the processor 580 for processing, and then transmit the audio data to another terminal through the RF circuit 510, or output the audio data to the memory 520 for further processing. The audio circuit 560 can further include an earphone jack to provide communication of an external earphone with the electronic device 500.

[0112] The electronic device 500 can help the user to receive requests, transmit information, and the like through the transmission module 570 (e.g., a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown, it can be understood that it does not belong to the essential components of the electronic device 500, and can be omitted as needed without changing the essence of the application.

[0113] The processor 580 is a control center of the electronic device 500 that uses various interfaces and lines to connect the entire mobile phone with various parts, performs various functions of the electronic device 500 and processes data by running or executing software programs and / or modules stored in the memory 520 and calling data stored in the memory 520, thereby overall monitoring the electronic device. Optionally, the processor 580 can include one or more processing cores; in some embodiments, the processor 580 can integrate an application processor and a modem processor, in which the application processor mainly processes an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 580.

[0114] The electronic device 500 further includes a power supply 590 (such as a battery) for supplying power to various components, and in some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 590 can also include one or more than one direct or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power state indicator, and any other components.

[0115] Although not shown, the electronic device 500 further includes a camera (such as a front camera or a rear camera), a Bluetooth module, and the like, which are not described here in detail. Specifically, in the present embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal further includes a memory and one or more than one program, wherein the one or more than one program is stored in the memory and is configured to be executed by the one or more than one processor, and the one or more than one program includes instructions for performing the following operations:

[0116] Obtaining a visual image sequence; the visual image sequence includes a plurality of visual images;

[0117] Synchronizing instance segmentation of the visual images, key frame extraction, feature extraction and matching of the key frames using a temporal communication mechanism, to obtain a feature matching pair;

[0118] Using a semantic auxiliary RANSAC method to remove static outliers from the feature matching pair, and identifying dynamic feature points in combination with instance information and Sampson geometric information;

[0119] Constructing an IMU pre-integration residual error and an edge residual error, removing the identified dynamic feature points, constructing a visual re-projection residual error using the remaining static feature points, and calculating an optimized pose through a solver.

[0120] In practice, the above modules can be implemented as independent entities, or combined as the same or several entities, and the specific implementation of the above modules can refer to the method embodiments above, which will not be repeated here.

[0121] Those skilled in the art can understand that all or part of the steps in the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor. Therefore, the embodiments of the present application provide a storage medium, which stores a plurality of instructions capable of being loaded by a processor to execute the steps of any embodiment of the visual-inertial SLAM method suitable for a complex dynamic environment provided by the embodiments of the present application.

[0122] The computer readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0123] Since the instructions stored in the storage medium can execute the steps in any embodiment of the visual-inertial SLAM method suitable for a complex dynamic environment provided by the embodiments of the present application, the beneficial effects of any visual-inertial SLAM method suitable for a complex dynamic environment provided by the embodiments of the present application can be achieved, which will be described in detail in the above embodiments, and will not be repeated here.

[0124] The above describes in detail the visual-inertial SLAM method suitable for a complex dynamic environment, system, storage medium and electronic device provided by the embodiments of the present application, and the principle and implementation manner of the present application are described by applying specific examples; the above embodiment is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A visual-inertial SLAM method suitable for complex dynamic environments, characterized in that, The method comprises: acquiring a visual image sequence; the visual image sequence comprises a plurality of visual images; synchronizing instance segmentation of the visual images, key frame extraction, feature extraction and matching of the key frames by using a time communication mechanism to obtain a feature matching pair, comprising: in a neural network thread, transmitting time information corresponding to a received visual image to a VIO thread, performing instance segmentation on the visual image to obtain instance pixels and instance IDs, constructing an instance mask based on the instance pixels and the instance IDs, and transmitting the instance mask to the VIO thread; in the VIO thread, extracting a corresponding visual image in the image sequence according to the time information, obtaining a feature matching pair by performing feature tracking on feature points at a previous time by using an optical flow method, eliminating tracking points that fail to track and exceed a boundary, and extracting new corner points as supplementary feature points; regarding gray pixels at corresponding positions in the instance mask as instance ID information, and associating and matching the instance ID information with corresponding feature points; performing static outlier elimination on the feature matching pair by using a semantic auxiliary RANSAC method, and identifying dynamic feature points in combination with instance information and Sampson geometric information; constructing an IMU pre-integration residual error and a marginalization residual error, eliminating the identified dynamic feature points, constructing a visual re-projection residual error by using remaining static feature points, and calculating an optimized pose by using a solver.

2. The visual-inertial SLAM method suitable for complex dynamic environments according to claim 1, wherein, The method comprises: constructing a gray image based on the instance pixels and corresponding instance IDs; performing instance ID coverage on the gray image according to a segmentation order for different instance pixel overlapping regions; regarding the covered gray image as an instance mask. 3.The visual-inertial SLAM method suitable for complex dynamic environments according to claim 1, wherein, The method comprises: regarding the feature matching pair as a prior point set, and eliminating feature points belonging to potential dynamic semantics in the feature matching pair; dividing the feature matching pair after eliminating the feature points belonging to potential dynamic semantics into different clusters according to instance ID information to which each feature belongs; calculating a fundamental matrix of adjacent key frames, eliminating outliers in the feature matching pair based on the fundamental matrix, and adding the feature points belonging to potential dynamic semantics that are eliminated before.

4. The visual-inertial SLAM method suitable for complex dynamic environments of claim 3, wherein, The method comprises: calculating epipolar error information of each instance object based on the fundamental matrix, and adding the epipolar error information to a geometric error sequence of the instance object; for each instance object, calculating a proportion of points in the geometric error sequence that are greater than a preset error threshold, and if the proportion is greater than 80%, determining that the instance object is in a motion state, and regarding all feature points belonging to the instance object as dynamic features.

5. The visual-inertial SLAM method suitable for complex dynamic environments of claim 4, wherein, The calculation formula of the epipolar error information is: wherein , are the homogeneous coordinates of the pixel coordinates in the adjacent keyframes and , respectively, is the fundamental matrix and T denotes the transpose.

6. The visual-inertial SLAM method suitable for complex dynamic environments of claim 1, wherein, The method comprises: Compute IMU pre-integration residual error of adjacent keyframes; Compute marginalization residual error according to marginalization information of sliding window; Iterate all feature points, eliminate identified dynamic feature points, and use remaining static feature points to construct visual re-projection residual error of initial observation frame and other observation frames; Solve residual function in solver using DogLeg method to obtain optimized pose.

7. A visual-inertial SLAM system suitable for complex dynamic environments for implementing the visual-inertial SLAM method suitable for complex dynamic environments as claimed in claim 1, characterized in that, The method comprises the following steps: An acquisition module is configured to acquire a visual image sequence, wherein the visual image sequence comprises a plurality of visual images. A keyframe elastic regulation module is configured to extract keyframes by instance segmentation and to perform feature extraction and matching on the keyframes by using a time communication mechanism to synchronize the visual images, so as to obtain a feature matching pair. A cascaded visual outlier elimination module is configured to eliminate static outlier points from the feature matching pair by using a semantic-assisted RANSAC method, and to identify dynamic feature points in combination with instance information and Sampson geometric information. A backend optimization module is configured to construct IMU pre-integration residual error and marginalization residual error, to eliminate identified dynamic feature points, to construct visual re-projection residual error by using remaining static feature points, and to calculate an optimized pose by using a solver.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, which are adapted to be loaded by a processor to execute the visual-inertial SLAM method suitable for a complex dynamic environment according to any one of claims 1 to 6.

9. An electronic device, comprising: The processor and the memory are electrically connected, the memory is configured to store instructions and data, and the processor is configured to execute the steps in the visual-inertial SLAM method suitable for a complex dynamic environment according to any one of claims 1 to 6.

Images