Method, system, electronic device and storage medium for detecting tracking status of VIO system

Abstract

The present application relates to a method, system, electronic device and storage medium for detecting the tracking state of a VIO system. Training data is obtained by annotating source input data and system internal state information, the source input data includes IMU data and original images, and the training data includes positive samples and negative samples; model training is performed on the training data according to a machine learning method to obtain a trained machine learning model, and the trained machine learning model is used for tracking state detection; data to be predicted is received, and the score of the data to be predicted is output through the trained machine learning model, and the first tracking state of the VIO system is obtained according to the score of the data to be predicted. This solution simultaneously utilizes the source input data of the VIO system and part of the system internal state information, and uses a machine learning method to determine the tracking state. The machine learning method can learn from a large number of positive and negative samples, thereby obtaining the ability to accurately detect the tracking state, thereby improving the accuracy of detecting the tracking state.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, system, electronic device and storage medium for detecting the tracking status of a VIO system. Background Art

[0002] The Visual Inertial Odometry (VIO) algorithm is an algorithm that integrates camera and inertial measurement unit (IMU) information to calculate the position of a device in space. With its high efficiency and robustness, it is widely used in fields such as augmented reality (AR) and unmanned vehicles. During operation, the VIO system may change its tracking state due to changes in the scene or its own motion mode. Accurately judging the tracking state of the VIO system is an extremely important part of maintaining the stable operation of the VIO system. In related technologies, the traditional tracking state detection method simply checks some variable parameters inside the VIO system and uses some general thresholds to judge the tracking state, resulting in low accuracy.

[0003] Currently, no effective solution has been proposed for the problem of low accuracy of traditional tracking state detection methods in related technologies. Summary of the invention

[0004] The embodiments of the present application provide a method, system, electronic device and storage medium for detecting the tracking status of a VIO system, so as to at least solve the problem of low accuracy of the traditional tracking status detection method in the related art.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting a tracking state of a VIO system, the method comprising:

[0006] The source input data and the system internal state information are labeled to obtain training data, wherein the source input data includes IMU data and original images, and the training data includes positive samples and negative samples, wherein the positive samples are samples with good tracking, and the negative samples are samples with failed tracking;

[0007] Performing model training on the training data according to a machine learning method to obtain a trained machine learning model, wherein the trained machine learning model is used for tracking state detection;

[0008] Receive data to be predicted, output the score of the data to be predicted through the trained machine learning model, and obtain the first tracking state of the VIO system according to the score of the data to be predicted.

[0009] In some embodiments, after receiving the data to be predicted and outputting the score of the data to be predicted by the trained machine learning model, the method further includes:

[0010] Acquire system internal state information, classify the system internal state information, score each category according to preset rules, and obtain a score for each category;

[0011] Obtaining a comprehensive score according to the scores of each category and a preset comprehensive scoring strategy, and determining a second tracking state of the VIO system according to the comprehensive score;

[0012] A third tracking state of the VIO system is obtained according to the first tracking state and the second tracking state.

[0013] In some embodiments, obtaining a comprehensive score according to the scores of each category and a preset comprehensive score strategy includes:

[0014] Get the minimum score, maximum score and average score of each category;

[0015] If the minimum score is less than the first preset value, the comprehensive score is the minimum score;

[0016] If the maximum score is greater than the second preset value, and the minimum score is greater than the third preset value, the comprehensive score is the average score;

[0017] Otherwise, the comprehensive score is the product of all classification scores that are less than the third preset value.

[0018] In some embodiments, after obtaining the first tracking state, the second tracking state or the third tracking state, the method further includes:

[0019] Different response strategies are triggered according to different tracking states, and the response strategies are used to adjust the VIO system, wherein the tracking state includes the first tracking state, the second tracking state or the third tracking state.

[0020] In some embodiments, triggering different response strategies according to different tracking states includes:

[0021] If the tracking state is tracking limited, the constraint of the visual camera on the VIO system is weakened, and the constraint of the IMU on the VIO system is strengthened;

[0022] If the tracking state is severely limited, degenerating the 6DOF of the VIO system to 3DOF;

[0023] If the tracking state is tracking failure, the VIO system is relocated. If the tracking state is still tracking failure after the relocation, the VIO system is restarted.

[0024] In some embodiments, after obtaining the system internal state information, the method further includes:

[0025] The system internal state information is stored in a sliding window manner.

[0026] In a second aspect, an embodiment of the present application provides a system for detecting a tracking state of a VIO system, the system comprising a machine learning module, the machine learning module comprising a labeling module, a training module and a detection module.

[0027] The labeling module is used to label the source input data and the system internal state information to obtain training data, wherein the source input data includes IMU data and original images, and the training data includes positive samples and negative samples, wherein the positive samples are samples with good tracking, and the negative samples are samples with failed tracking;

[0028] The training module is used to perform model training on the training data according to a machine learning method to obtain a trained machine learning model, wherein the trained machine learning model is used for tracking state detection;

[0029] The detection module is used to receive the data to be predicted, output the score of the data to be predicted through the trained machine learning model, and obtain the first tracking state of the VIO system according to the score of the data to be predicted.

[0030] In some embodiments, the system further includes a system internal state information detection module and a comprehensive judgment module. After the detection module outputs the score of the data to be predicted,

[0031] The internal state information detection module obtains the system internal state information, classifies the system internal state information, scores each category according to a preset rule, and obtains a score for each category;

[0032] According to the scores of each category and the preset comprehensive scoring strategy, a comprehensive score is obtained, and the second tracking state of the VIO system is determined according to the comprehensive score.

[0033] The comprehensive judgment module obtains a third tracking state of the VIO system according to the first tracking state and the second tracking state.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for detecting the tracking status of a VIO system as described in the first aspect above is implemented.

[0035] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for detecting the tracking status of a VIO system as described in the first aspect above.

[0036] Compared with the related art, the method for detecting the tracking state of the VIO system provided in the embodiment of the present application obtains training data by annotating source input data and system internal state information, wherein the source input data includes IMU data and original images, and the training data includes positive samples and negative samples, the positive samples are samples with good tracking, and the negative samples are samples with failed tracking; the training data is subjected to model training according to a machine learning method to obtain a trained machine learning model, wherein the trained machine learning model is used for tracking state detection; data to be predicted is received, and the score of the data to be predicted is output through the trained machine learning model, and the first tracking state of the VIO system is obtained according to the score of the data to be predicted, and the source input data of the VIO system and part of the system internal state information are utilized at the same time, and a machine learning method is used to judge the tracking state, and the machine learning method can learn from a large number of positive and negative samples, thereby obtaining the ability to accurately detect the tracking state, thereby improving the accuracy of detecting the tracking state. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 is a flowchart of a method for detecting a tracking state of a VIO system according to an embodiment of the present application;

[0039] Figure 2 is a structural block diagram of a system for detecting a tracking state of a VIO system according to an embodiment of the present application;

[0040] Figure 3 It is a structural block diagram of another system for detecting the tracking status of a VIO system according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0042] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0043] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0044] This embodiment provides a method for detecting the tracking status of a VIO system. Figure 1is a flow chart of a method for detecting a tracking state of a VIO system according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0045] Step S101, annotating source input data and system internal state information to obtain training data, wherein the source input data includes IMU data and original images, and the training data includes positive samples and negative samples, where the positive samples are samples with good tracking and the negative samples are samples with failed tracking;

[0046] In this embodiment, the source input data of the system and part of the internal state information of the system are derived from the operation process of the VIO system. The source input data is the Raw Data (unprocessed data) of the visual camera and IMU. Part of the internal state information of the system refers to the information such as parallax, speed, and number of extracted feature points that are initially processed by the internal algorithm of the system. This information belongs to the internal information of the system and is directly derived from the Raw Data. It can be regarded as the refinement of the Raw Data, so it has a high training value. The tracking points of the system are the result of the deep operation and processing of the system and cannot be regarded as the refinement of the Raw Data, so it does not participate in labeling and training.

[0047] Based on the source input data and a part of the system's internal state information, the changes in the scene, the changes in the motion mode and the performance of the VIO system during operation are obtained. Among them, the performance of the VIO system during operation is a result performance, which is the result of the scene change or the motion mode change on the VIO system. For example, a white wall scene will result in the failure to extract feature points, which will cause the VIO system to shake violently or even fly away due to the lack of visual information constraints. For example, pure rotation will cause the VIO system to be unable to triangulate map points or the triangulated map points to be in the wrong position, causing the VIO system trajectory to shake or fly away. However, scenes with rich textures and correct motion modes will allow the VIO system to run stably, and the motion trajectory will be smooth without jumps.

[0048] The data is labeled according to the changes in the scene, the changes in the motion pattern, and the performance of the VIO system during operation. For example, if a certain section of data is a white wall scene, the running result of this section of data is labeled as white wall; if a certain section of data is a scene with too intense motion, the running result of this section of data is labeled as too intense motion pattern. Data with poor tracking are uniformly labeled as negative samples, and data with good tracking are labeled as positive samples.

[0049] Step S102, training the training data according to the machine learning method to obtain a trained machine learning model, wherein the machine learning model is used for tracking state detection; in this embodiment, machine learning methods such as support vector machine (SVM) algorithm, naive Bayesian classifier (NBC) algorithm, logistic regression (LR) algorithm, K-nearest neighbor (KNN) algorithm or artificial neural network (ANN) algorithm can be used for model training. The machine learning method can learn from a large number of positive and negative samples, thereby obtaining the ability to accurately detect the tracking state of the VIO system.

[0050] Step S103, receiving the data to be predicted, outputting the score of the data to be predicted through the trained machine learning model, and obtaining the first tracking state of the VIO system according to the score of the data to be predicted. In this embodiment, the data to be predicted is the source input data and a part of the internal state information of the system. When the score is greater than the first preset threshold, the first tracking state is good tracking, and when the score is between the first preset threshold and the second preset threshold, the first tracking state is limited tracking, and when the score is less than the second preset threshold, the first tracking state is tracking failure, wherein the first preset threshold is greater than the second preset threshold.

[0051] Compared with the related art, the traditional tracking state detection method performs a simple check on some variable parameters inside the VIO system and uses some general thresholds to judge the tracking state. For example, the number of tracked points or parallax is used to simply judge the tracking state. This detection method cannot utilize the source input data of the VIO system and the utilization of the system's internal state information is very limited, resulting in low accuracy of the detection result. In the technical solution of the present application, through steps S101 to S103, the source input data of the VIO system and part of the system's internal state information are simultaneously utilized, and a machine learning method is used to judge the tracking state. The machine learning method can learn from a large number of positive and negative samples, thereby obtaining the ability to accurately detect the tracking state, thereby improving the accuracy of detecting the tracking state.

[0052] In some of the embodiments, after receiving the data to be predicted and outputting the score of the data to be predicted through the machine learning model, the internal state information of the system is obtained, the internal state information of the system is classified, and each classification is scored according to preset rules to obtain the score of each classification; a comprehensive score is obtained based on the score of each classification and a preset comprehensive scoring strategy, and the second tracking state of the VIO system is determined based on the comprehensive score; and the third tracking state of the VIO system is obtained based on the first tracking state and the second tracking state.

[0053] Exemplarily, the internal state information of the system can be divided into 12 categories, namely, the IMU acceleration information acc corrected within the system, the IMU angular velocity information gyr corrected within the system, the velocity velocity calculated by the system according to the distance between two frames and the timestamp between the two frames, the distance position_diff between the two frames, the rotation angle orientation_diff between the two frames, the image texture richness texture_richness (calculated by the number of roughly extracted fast corner points), the image distribution uniformity feature_distribution (dividing the original image into 4x4=16 equal parts of the grid grid, and counting the number of fast corner points in each grid), parallax (obtained by the difference in the number of pixels between two features obtained by optical flow matching), parallax variance parallax_covariance (describing the uniformity of parallax distribution), the number of points successfully tracked by the front end frontend_tracks, the number of points successfully tracked by the back end after processing backend_tracks, and the scene depth scene_depth (the deeper the scene, the lower the score). After the internal state information of the system is classified, the above categories are roughly scored as follows:

[0054] 1.Acc_score_raw=acc_rectify.norm() / 4.0;

[0055] 2.Gyr_score_raw=gyr_rectify.norm() / 0.8;

[0056] 3.Velocity_score_raw=velocity.norm() / 4.0;

[0057] 4.Position_diff_score_raw=position_diff.norm() / 0.1;

[0058] 5.Orientation_diff_score_raw=orientation_diff.norm() / 0.5;

[0059] 6.Texture_richness_score_raw=source_features_num / 1500;

[0060]

[0061] 8.Parallax_score_raw=average_parallax / 4.0;

[0062]

[0063] 10.Frontend_tracks_score_raw=frontend_tracked_features / 80;

[0064] 11.Backend_tracks_score_raw=backend_tracked_features / 20;

[0065] 12.Scene_depth_score_raw=average_inverse_depth_of_backend_tracked_features / 0.2;

[0066] Among them, the norm function is the norm of the vector. After rough scoring, 8 activation functions are used to convert linear scores into nonlinear scores to better fit the actual situation. The 8 basic activation functions are as follows:

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075] The 8 activation functions convert the linear scores into nonlinear scores as shown below (full score is 1, and the score range is [0,1]):

[0076] 1.acc_score=Sigmoid_acc(Acc_score_raw);

[0077] 2.gyr_score=Sigmoid_acc(Gyr_score_raw);

[0078] 3.velocity_score=Sigmoid_velocity(Velocity_score_raw);

[0079] 4.position_diff_score=Sigmoid_pose(Position_diff_score_raw);

[0080] 5.orientation_diff_score=Sigmoid_pose(Orientation_diff_score_raw);

[0081] 6.texture_richness_score=Sigmoid_feature(Texture_richness_score_raw);

[0082] 7.feature_distribution_score=Sigmoid_feature(Features_distribution_score_raw);

[0083] 8.parallax_score=Sigmoid_parallax(Parallax_score_raw);

[0084] 9.parallax_covariance_score=Sigmoid_parallax(Parallax_covariance_score_raw);

[0085] 10.frontend_tracks_score=Sigmoid_tracking(Frontend_tracks_score_raw);

[0086] 11.backend_tracks_score=Sigmoid_tracking(Backend_tracks_score_raw);

[0087] 12.scene_depth_score=Sigmoid_depth(Scene_depth_score_raw);

[0088] After obtaining the scores of each category through the above 12 formulas, a comprehensive score is obtained according to the weight of the scores of each category or a preset comprehensive scoring strategy. If the comprehensive score is greater than the first preset threshold, the second tracking state is good tracking; if the comprehensive score is between the first preset threshold and the second preset threshold, the second tracking state is restricted tracking; if the comprehensive score is less than the second preset threshold, the second tracking state is tracking failure, wherein the first preset threshold is greater than the second preset threshold.

[0089] In practical applications, the VIO tracking state can be judged according to the first tracking state or the second tracking state. The first tracking state is obtained by machine learning methods. The advantage is that it can learn certain specific scenes and motion patterns, and can quickly and carefully judge when encountering such scenes or motion patterns. The disadvantage is that it lacks the support of the system's internal state information, so it cannot achieve a very high degree of accuracy. For example, when the texture is relatively missing or the motion pattern is relatively abnormal, the machine learning method lacks the support of the system's internal state information (such as the number of tracking points), and the judgment accuracy of these situations will not be very high. It may be better or worse to track. The second tracking state is obtained by the method of judging the system's internal state information. The advantage is that it can directly judge the quality of the VIO system's tracking state. The disadvantage is that it cannot accurately judge which factor causes the good or bad tracking.

[0090] In order to solve the above problems, the two judgment methods are combined, and the third tracking state of the VIO system is obtained according to the first tracking state and the second tracking state to further improve the accuracy of detecting the tracking state. The third tracking state of the VIO system is obtained according to the first tracking state and the second tracking state, including: when the first tracking state and the second tracking state are both tracking well, the third tracking state is also tracking well; when at least one of the first tracking state and the second tracking state is tracking restricted, and neither is tracking failed, the third tracking state is tracking restricted; when at least one of the first tracking state and the second tracking state is tracking failed, and neither is tracking good, the third tracking state is tracking failed.

[0091] If there are special circumstances, for example, the first tracking state is good tracking and the second tracking state is tracking failure, check whether the velocity_score and backend_tracks_score of the system internal status information are both less than 0.1. If so, the third tracking state is tracking failure; if not, the third tracking state is tracking restricted. That is, the problem of inconsistent judgments between the first tracking state and the second tracking state is solved by using some individual scores.

[0092] Typical scenes and motion mode judgment:

[0093] White-wall scene judgment: The machine learning method judges that the score of the white-wall scene is high. At this time, if the texture_richness_score and feature_distribution_score scores in the system internal state information judgment method are both less than 0.2, it is officially judged as a white-wall scene;

[0094] Depth scene judgment: The machine learning method judges that the current scene is a deep scene with a high score. At this time, if the scene_depth_score score in the system internal state information judgment method is less than 0.2 points, it is officially judged as a deep scene;

[0095] Fast motion mode judgment: The machine learning method judges that the fast motion score is high. At this time, if the velocity_score, parallax_score and parallax_covariance_score scores in the system internal state information judgment method are all less than 0.2, and the backend_tracks_score is less than 0.3, it is judged as a fast motion mode;

[0096] Pure rotation mode judgment: The machine learning method judges that the score of pure rotation motion is relatively high. At this time, if the gyr_score score in the internal state information judgment method is less than 0.1, the parallax_score and parallax_covariance_score scores are both less than 0.2, and the velocity_score and position_diff_score scores are both greater than 0.85, then it is judged as a pure rotation motion mode. Some other mode joint judgment schemes are no longer repeated. Through the joint judgment of the two methods, the shortcomings of the two methods themselves are well compensated, thereby achieving the purpose of accurately judging the tracking state of the VIO system.

[0097] In some embodiments, obtaining a comprehensive score according to the scores of each category and a preset comprehensive score strategy includes:

[0098] Get the minimum score min_score, maximum score max_score and average score ave_score of each category score; if the minimum score is less than the first preset value, the comprehensive score is the minimum score; if the maximum score is greater than the second preset value, and the minimum score is greater than the third preset value, the comprehensive score is the average score; otherwise, the comprehensive score is the product of all category scores that are less than the third preset value.

[0099] For example, the first preset value is 0.1, the second preset value is 0.9, and the third preset value is 0.4, then the comprehensive scoring strategy is:

[0100] min_score = min(acc_score, gyr_score, velocity_score, position_diff_score, orientation_diff_score, texture_richness_score, feature_distribution_score, parallax_score, parallax_covariance_score, frontend_tracks_score, backend_tracks_score, scene_depth_score);

[0101] max_score = max(acc_score, gyr_score, velocity_score, position_diff_score, orientation_diff_score, texture_richness_score, feature_distribution_score, parallax_score, parallax_covariance_score, frontend_tracks_score, backend_tracks_score, scene_depth_score);

[0102] ave_score = sum(acc_score, gyr_score, velocity_score, position_diff_score, orientation_diff_score, image_richness_score, feature_distribution_score, parallax_score, parallax_covariance_score, frontend_tracks_score, backend_tracks_score, scene_depth_score) / 12;

[0103] if min_score < 0.1: score = min_score; # If the minimum score is less than 0.1, the comprehensive score is the minimum score

[0104] else if max_score>0.9&&min_score>0.4:score=ave_score;#If the maximum score is greater than 0.9 and the minimum score is greater than 0.4, the comprehensive score is the average score

[0105] else:

[0106] score=times_if_score_less_than_0.4(acc_score,gyr_score,velocity_score,position_diff_score,orientation_diff_score,texture_richness_score,feature_distribution_score,parallax_score,parallax_covariance_score,frontend_tracks_score,backend_tracks_score,scene_depth_score);

[0107] #times_if_score_less_than_0.4 means multiply all scores less than 0.4.

[0108] In some of the embodiments, after obtaining the first tracking state, the second tracking state or the third tracking state, different response strategies are triggered according to different tracking states, and the response strategies are used to adjust the VIO system, wherein the tracking state includes the first tracking state, the second tracking state or the third tracking state.

[0109] Optionally, the tracking status of Tracking Restriction can be further divided into Tracking Restriction and Tracking Restriction Severe. Different response strategies are triggered according to different tracking statuses, including:

[0110] If the tracking status is tracking well, the VIO system is operating normally;

[0111] If the tracking state is tracking limited, the constraint of the visual camera on the VIO system is weakened, and the constraint of the IMU on the VIO system is strengthened. In this embodiment, the tracking limitation may occur because the texture is poor or the motion pattern is poor. At this time, the visual feature points extracted by the VIO system are difficult to maintain the stable operation of the system. After accurately judging this situation, it is chosen to weaken the constraint of the visual camera on the VIO system and strengthen the constraint of the IMU on the VIO system (that is, trust the observation of the IMU more, because the IMU can obtain more reliable observations than vision when it is in rapid motion or pure rotation), thereby reducing the adverse effects of the tracking limitation state on the VIO system and improving the robustness of the VIO system.

[0112] If the tracking status is severely limited, the 6DOF of the VIO system is degraded to 3DOF. In this embodiment, the tracking is severely limited, that is, the texture is severely missing and the motion mode is very unfriendly, but the VIO system does not fly away. The 6DOF (6 Degree of Freedom), that is, 3 displacement degrees of freedom + 3 rotation degrees of freedom, is degraded to 3DOF, that is, 3 rotation degrees of freedom for processing. Because the IMU's observation of rotation is more reliable and has a higher confidence level, the rotation information provided by the IMU can basically be trusted. After the VIO system is degraded to 3DOF in the AR scene, the video display of the three-dimensional AR object can still be seen, but at this time the AR object will only rotate and not displace, but the user can still see that the AR object will not disappear in the scene.

[0113] If the tracking state is tracking failure, the VIO system is repositioned. If the tracking state is still tracking failure after repositioning, the VIO system is restarted. In this embodiment, tracking failure means that the tracking trajectory of the VIO system has a serious jump or even flies away. The VIO system is pulled back to a normal trajectory by repositioning. When repositioning does not work, the VIO system is restarted to restart the VIO system to prevent the deterioration of the VIO system state.

[0114] The above situations are all handled based on the accurate judgment of the tracking status of the VIO system. If the tracking status of the VIO system is judged incorrectly, it may bring disastrous consequences to the VIO system. For example, good tracking is judged as limited tracking or tracking failure, and the originally good visual observation may be replaced by unstable IMU observation, resulting in reduced tracking accuracy or even system restart; if limited tracking is misjudged as good tracking, the strategy that could have been used to compensate cannot be implemented, resulting in the tracking accuracy being damaged by erroneous observations; if tracking failure is misjudged as limited tracking or even good tracking, the algorithm cannot be restarted, resulting in tracking flying away, the user cannot see the AR object, and the user experience of the VIO system is extremely poor. Therefore, accurate detection of the tracking status can maintain the stable operation of the VIO system.

[0115] In some embodiments, after obtaining the system internal state information, the system internal state information is stored in a sliding window manner. The sliding window has higher robustness than a single frame. For example, a single frame may give some abnormal data due to an abnormality of the IMU sensor, thereby causing an error in judgment. However, the probability of an error in the entire sliding window is very small and almost impossible. If it occurs, it is likely that the IMU sensor is damaged.

[0116] It should be noted that the VIO algorithm in the present application scheme can be replaced by VIO algorithms such as ROVIO and OKVIS, and the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and, although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0117] This embodiment also provides a system for detecting the tracking status of a VIO system, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0118] Figure 2 is a structural block diagram of a system for detecting the tracking state of a VIO system according to an embodiment of the present application, such as Figure 2 As shown, the system includes a machine learning module 21, and the machine learning module 21 includes a labeling module 211, a training module 212 and a detection module 213. The labeling module 211 is used to label the source input data and the internal state information of the system to obtain the training data, and the source input data includes IMU data and original images. The training data includes positive samples and negative samples, wherein the positive samples are samples with good tracking and the negative samples are samples with failed tracking; the training module 212 is used to perform model training on the training data according to the machine learning method to obtain a trained machine learning model, wherein the trained machine learning model is used for tracking state detection; the detection module 213 is used to receive the data to be predicted, output the score of the data to be predicted through the trained machine learning model, and obtain the first tracking state of the VIO system according to the score of the data to be predicted.

[0119] In some of these embodiments, Figure 3 is a structural block diagram of another system for detecting the tracking state of a VIO system according to an embodiment of the present application, such as Figure 3 As shown, the system also includes a system internal state information detection module 31 and a comprehensive judgment module 32. After the detection module 213 outputs the score of the data to be predicted, the internal state information detection module 31 obtains the system internal state information, classifies the system internal state information, scores each category according to preset rules, and obtains the score of each category; obtains a comprehensive score based on the score of each category and a preset comprehensive scoring strategy, and judges the second tracking state of the VIO system based on the comprehensive score. The comprehensive judgment module 32 obtains the third tracking state of the VIO system based on the first tracking state and the second tracking state.

[0120] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0121] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0122] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0123] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0124] In addition, in combination with the method for detecting the tracking state of the VIO system in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the methods for detecting the tracking state of the VIO system in the above embodiment is implemented.

[0125] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for detecting the tracking state of a VIO system is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball, or touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0126] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0127] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for detecting the tracking state of a VIO system, characterized in that: The method comprises: The source input data and the system internal state information are labeled to obtain training data, wherein the source input data includes IMU data and original images, and the system internal state information refers to the information of Raw Data related to parallax, speed, and number of extracted feature points, which is preliminarily processed by the system internal algorithm, and belongs to the internal information of the system and is directly derived from the Raw Data; the training data includes positive samples and negative samples, the positive samples are samples with good tracking, and the negative samples are samples with failed tracking; Performing model training on the training data according to a machine learning method to obtain a trained machine learning model, wherein the trained machine learning model is used for tracking state detection, and the tracking state includes tracking limited, tracking severely limited, tracking failed, and tracking well; receiving data to be predicted, outputting scores of the data to be predicted through the trained machine learning model, and obtaining a first tracking state of the VIO system according to the scores of the data to be predicted; Wherein, after receiving the data to be predicted and outputting the score of the data to be predicted by the trained machine learning model, the method further includes: Acquire system internal state information, classify the system internal state information, score each category according to preset rules, and obtain a score for each category; Obtaining a comprehensive score according to the scores of each category and a preset comprehensive scoring strategy, and determining a second tracking state of the VIO system according to the comprehensive score; A third tracking state of the VIO system is obtained according to the first tracking state and the second tracking state.

2. The method according to claim 1, characterized in that According to the scores of each category and the preset comprehensive scoring strategy, the comprehensive score is obtained including: Get the minimum score, maximum score and average score of each category; If the minimum score is less than the first preset value, the comprehensive score is the minimum score; If the maximum score is greater than the second preset value, and the minimum score is greater than the third preset value, the comprehensive score is the average score; Otherwise, the comprehensive score is the product of all classification scores that are less than the third preset value.

3. The method according to claim 1, characterized in that After obtaining the first tracking state, the second tracking state or the third tracking state, the method further includes: Different response strategies are triggered according to different tracking states, and the response strategies are used to adjust the VIO system, wherein the tracking state includes the first tracking state, the second tracking state or the third tracking state.

4. The method according to claim 3, characterized in that The triggering of different response strategies according to different tracking states includes: If the tracking state is tracking limited, the constraint of the visual camera on the VIO system is weakened, and the constraint of the IMU on the VIO system is strengthened; If the tracking state is severely limited, degenerating the 6DOF of the VIO system to 3DOF; If the tracking state is tracking failure, the VIO system is relocated. If the tracking state is still tracking failure after the relocation, the VIO system is restarted.

5. The method according to claim 1, characterized in that After obtaining the system internal state information, the method further includes: The system internal state information is stored in a sliding window manner.

6. A system for detecting the tracking status of a VIO system, characterized in that: The system includes a machine learning module, which includes a labeling module, a training module and a detection module. The labeling module is used to label the source input data and the system internal state information to obtain training data, wherein the source input data includes IMU data and original images, and the system internal state information refers to the information of Raw Data related to parallax, speed, and number of extracted feature points, which is preliminarily processed by the system internal algorithm, and belongs to the system internal information and is directly derived from RawData; the training data includes positive samples and negative samples, wherein the positive samples are samples with good tracking, and the negative samples are samples with failed tracking; The training module is used to perform model training on the training data according to a machine learning method to obtain a trained machine learning model, wherein the trained machine learning model is used for tracking state detection, and the tracking state includes tracking limited, tracking severely limited, tracking failed, and tracking well; The detection module is used to receive the data to be predicted, output the score of the data to be predicted through the trained machine learning model, and obtain the first tracking state of the VIO system according to the score of the data to be predicted; The system further includes a system internal state information detection module and a comprehensive judgment module. After the detection module outputs the score of the data to be predicted, the internal state information detection module obtains the system internal state information, classifies the system internal state information, scores each category according to a preset rule, and obtains the score of each category. Obtaining a comprehensive score according to the scores of each category and a preset comprehensive scoring strategy, and determining a second tracking state of the VIO system according to the comprehensive score; The comprehensive judgment module obtains a third tracking state of the VIO system according to the first tracking state and the second tracking state.

7. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for detecting the tracking state of a VIO system according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method for detecting the tracking state of a VIO system according to any one of claims 1 to 5 when running.

Images