Failure detection methods and systems for tracking systems

Abstract

This application relates to a failure detection method for a tracking system. The method includes: acquiring current image information through a processing module, and performing environmental anomaly detection based on the current image information, including camera lens occlusion detection, low-light environment detection, and large depth-of-field environment detection; acquiring current IMU information through the processing module, and performing user abnormal behavior anomaly detection based on the current IMU information, including drop detection and impact detection; and outputting corresponding abnormal state responses based on the different types of environmental anomaly detection results and user abnormal behavior detection results, instructing the AR device to perform appropriate anomaly protection. This application solves the problem of poor user experience caused by the lack of failure detection in related technologies, and provides a universal and reliable method for detecting and protecting against tracking failures.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of augmented reality, and in particular to a failure detection method and system for a tracking system. BACKGROUND

[0002] Pose tracking is a core basic function of head-mounted devices such as AR glasses, through which current pose data (6DoF) can be obtained; having 6DoF tracking capability is a prerequisite for AR devices to display real-time virtual-real combined scenes.

[0003] Failure of the tracking system, i.e., the 6DoF obtained by tracking no longer matches the actual movement of the user; the causes include environmental interference and user behavior, etc., and if the output result is continued to be based on the erroneous 6DoF tracking data, the virtual-real superimposed content displayed in the user interface will not match the actual situation.

[0004] In related technologies, existing AR devices lack a general and reliable detection and protection mechanism for tracking failure state. When the tracking system of an AR device fails (for example, the user accidentally blocks the camera carried by the AR glasses), the display of AR content will be immediately stopped until the pose tracking system recovers.

[0005] The above-mentioned manner will cause the user to feel confused, making the AR experience process fragmented and seriously affecting the user's experience. SUMMARY

[0006] Embodiments of the present application provide a tracking system failure detection method, system, computer device and computer readable storage medium to at least solve the problem of poor user experience caused by lack of failure detection of the tracking system in related technologies.

[0007] In some embodiments, the present application provides a tracking system failure detection method, characterized in that it is applied to an AR device and integrates a multi-dimensional abnormality detection function through an operation processing module, and the method comprises:

[0008] Through the operation processing module, current image information is obtained, and environmental abnormality detection is performed according to the current image information, wherein the environmental abnormality detection includes camera lens blocking detection, dark environment detection and large depth of field environment detection;

[0009] Through the operation processing module, current IMU information is obtained, and user abnormal behavior abnormality detection is performed according to the current IMU information, including drop detection and impact detection;

[0010] Through the operation processing module, corresponding abnormal state responses are output according to different types of environmental abnormality detection results and user abnormal behavior detection results, instructing the AR device to perform corresponding abnormal protection.

[0011] In some embodiments, the camera lens occlusion detection according to the current image information comprises:

[0012] Obtaining the key points of the current image, and obtaining the response intensity and distribution uniformity of the key points;

[0013] Judging whether the current response intensity is in a first preset response intensity interval, if yes, outputting an abnormal state response that the current camera lens is occluded;

[0014] Judging whether the current distribution uniformity is in a preset uniformity interval, if yes, outputting an abnormal state response that the current camera lens is occluded.

[0015] In some embodiments, the dark light environment detection according to the current image information comprises:

[0016] Obtaining the key points of the current image, and obtaining the response intensity of the key points;

[0017] Judging whether the response intensity of the key points is in a second preset response intensity interval, if yes, outputting an abnormal state response that the AR device is currently in a dark light environment.

[0018] In some embodiments, the large depth of field environment monitoring according to the current image information comprises:

[0019] Obtaining the length and average disparity of the tracked key points of the continuous image frames;

[0020] According to the length and average disparity, judging whether the current camera depth of field is much larger than the relative displacement of the camera, if yes, outputting an abnormal state response that the current camera is facing a large depth of field environment.

[0021] In some embodiments, the user abnormal behavior anomaly detection according to the current IMU information comprises:

[0022] According to the current IMU information, obtaining the acceleration change amount, judging whether the acceleration change amount is greater than a first change threshold, if yes, outputting an abnormal state response that the AR device is hit or dropped.

[0023] In some embodiments, the operation processing module, according to different types of environment anomaly detection results and user abnormal behavior detection results, comprises:

[0024] In the case that the abnormal state response is that the current camera lens is occluded or the AR device is currently in a dark light environment, instructing the AR device to perform 3DoF update;

[0025] In the case that the abnormal state response is that the current camera is facing a large depth of field environment, instructing the AR device to perform zero speed update;

[0026] In the case that the abnormal state response is that the AR device is hit or dropped, the AR device is instructed to perform abnormal speed value filtering.

[0027] In some embodiments, after the corresponding abnormal state response is output, indicating that the AR device performs corresponding abnormal protection, the method further comprises:

[0028] Continuously detecting whether the current AR device triggers any one of the following conditions: the camera is blocked, the AR device is currently in a dark environment, and the current camera is currently facing a large depth of field environment, and if not, switching the pose data to 6Dof.

[0029] In some embodiments, the system is applied on an AR device, and the system integrates multi-dimensional abnormal detection functions, including an image information processing module, an IMU information processing module, and a response instruction module, wherein:

[0030] The image processing module is configured to obtain current image information, and perform environment abnormal detection according to the current image information, wherein the environment abnormal detection includes camera lens blocking detection, dark environment detection, and large depth of field environment detection.

[0031] The IMU information processing module is configured to obtain current IMU information, and perform user abnormal behavior abnormal detection according to the current IMU information, including drop detection and hit detection.

[0032] The response instruction module is configured to output a corresponding abnormal state response according to different types of environment abnormal detection results and user abnormal behavior detection results, and instruct the AR device to perform corresponding abnormal protection.

[0033] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of the first aspect when executing the computer program.

[0034] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method of the first aspect.

[0035] Compared with the related art, the tracking system failure detection method provided by the embodiment of the application, through the operation processing module, obtains current image information, and performs environment anomaly detection according to the current image information, the environment anomaly detection including camera lens blocking detection, dark light environment detection and large depth of field environment detection; the current IMU information is taken, and user abnormal behavior anomaly detection is performed according to the current IMU information, including fall detection and impact detection; according to the environment anomaly detection result and the user abnormal behavior detection result of different types, the corresponding abnormal state response is output, indicating the AR device to perform corresponding abnormal protection. The problem that the tracking system in the related art lacks failure detection, resulting in poor user experience, is solved, a general and reliable detection and protection method for tracking failure state is provided, from the two dimensions of the environment where the AR device is located and the user behavior mode, based on sensor information, the common abnormal use mode of the AR device is monitored in all directions, and the corresponding multi-dimensional prompt signal is generated, and the corresponding abnormal protection is performed. It has important guiding value for improving the user experience of the AR device (especially the AR glasses). BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings described herein are used to provide further understanding of the application, and form a part of the application. The illustrative embodiments of the application and their descriptions are used to explain the application, and do not constitute an improper limitation on the application. In the drawings:

[0037] Figure 1 is a schematic diagram of an application environment of a tracking system failure detection method according to an embodiment of the application;

[0038] Figure 2 is a flowchart of a tracking system failure detection method according to an embodiment of the application;

[0039] Figure 3 is a structural block diagram of a tracking system failure detection system according to an embodiment of the application;

[0040] Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the application more clear, the application is described and explained below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and do not limit the application. Based on the embodiments provided by the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the application.

[0042] It is apparent that the accompanying drawings described below in the description are only some examples or embodiments of the present application, and for those skilled in the art, the present application can be applied to other similar situations according to the drawings without creative labor. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, some design, manufacture or production changes based on the technology disclosed in the present application are only routine technical means for those skilled in the art related to the disclosure of the present application, and should not be understood as insufficient disclosure of the present application.

[0043] In the present application, the term "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0044] Unless otherwise defined, the technical terms or scientific terms involved in the present application should be understood as the usual meaning understood by those skilled in the art in the technical field to which the present application belongs. The terms "one", "a", "an", "the", and similar words involved in the present application do not represent quantity limitation, but can represent singular or plural. The terms "include", "contain", "have", and any variations thereof involved in the present application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device including a series of steps or modules (units) is not limited to the listed steps or units, but can also include steps or units not listed, or can also include other steps or units inherent to the process, method, product or device. The terms "connected", "connected", "coupled" and similar words involved in the present application are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The term "multiple" refers to two or more. The association between the associated objects is described by the term "and / or", which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects. The terms "first", "second", "third", and the like involved in the present application are only to distinguish similar objects, and do not represent a specific order for the objects.

[0045] In this document, it should be understood that the terms involved can be technical means or other summary technical terms for implementing part of the present application, for example, the terms can include:

[0046] AR(Argument Reality): Augmented Reality, a technology that skillfully combines virtual information with the real world, simulates virtual information such as computer-generated text, images, three-dimensional models, and videos, and applies them to the real world, the two kinds of information complement each other, and realize the "augmentation" of the real world.

[0047] Pose: position and orientation (direction), for example: in two dimensions, it is generally (x, y, yaw), and in three dimensions, it is generally (x, y, z, yaw, pitch, roll), including 6 degrees of freedom, 6of (6 Degrees Of Freedom), the last three elements describe the attitude of the object, where yaw is the heading angle, rotating around the Z axis, pitch is the pitch angle, rotating around the Y axis, and roll is the roll angle, rotating around the X axis.

[0048] Large depth of field environment: for example, sky, lake surface, etc., a scene with a larger clear range and higher openness in the picture,

[0049] Image gradient: refers to the rate of change of a pixel in an image in x and y directions (compared with adjacent pixels), which is a two-dimensional vector composed of two components, X-axis change and Y-axis change.

[0050] The tracking system failure detection method provided in the application can be applied in the application environment as shown in Figure 1 The application environment of the tracking system failure detection method according to the embodiment of the application is shown in Figure 1 As shown in Figure 1 The user operates the terminal device 10 in an offline scene to experience AR. When the terminal device 10 receives an impact or is in an abnormal state such as a lens being blocked, the tracking faces failure, the internal operation processing module can detect various abnormal behaviors in a general and reliable manner, and can be targeted for protection, thereby improving the user's experience. It should be noted that the terminal device 10 described above can be AR glasses, a smart phone, a tablet computer, a notebook computer, etc.

[0051] Figure 2 The flowchart of the tracking system failure detection method according to the embodiment of the application is shown in Figure 2 The flowchart of the tracking system failure detection method according to the embodiment of the application is shown in

[0052] S201, acquiring current image information by an operation processing module, and performing environment anomaly detection according to the current image information, the environment anomaly detection including camera lens blocking detection, dark environment detection, and large depth of field environment detection;

[0053] In this embodiment, the operation processing module integrates multiple dimensional anomaly detection functions, which can detect whether the tracking system fails from multiple aspects such as user behavior and light. The method provided in this embodiment is applied to an AR device with tracking function, which can be AR glasses, or a terminal device such as a smartphone and a tablet computer with AR experience function.

[0054] It should be noted that the pose tracking function is the core basic function of AR glasses. The AR device superimposes virtual content on the real scene according to real-time pose data. The pose tracking is a prerequisite for the AR device to display a virtual superimposed scene in real time.

[0055] Part of the abnormal situation will cause the tracking system of the AR device to fail. This abnormal situation can be caused by environmental factors, for example, the camera lens is blocked, or the camera is in a dark light environment; or caused by user behavior, for example, the device is hit or falls from a high place.

[0056] Among them, for the tracking failure caused by environmental factors, the operation processing module can judge through image information. Specifically, whether the camera lens is blocked or in a dark light environment can be judged according to the response strength and distribution rule of the image key point;

[0057] In addition, a large depth of field environment will also cause the tracking system to fail. In this embodiment, the operation processing module identifies whether the field of view of the current AR device is towards a large depth of field environment by judging the length of key point tracking between consecutive image frames and the average disparity.

[0058] S202, through the operation processing module, the current IMU information is acquired and user abnormal behavior anomaly detection is performed according to the current IMU information, including: fall detection and impact detection;

[0059] User behavior such as vibration, falling and other extreme factors may also cause 6DoF tracking to fail, resulting in the inability to continuously provide 6DoF tracking capability. For tracking system failure caused by user behavior, the change of acceleration in the IMU (Inertial Measurement Unit) can be used to realize detection;

[0060] Specifically, a certain acceleration change threshold can be set. By comparing the real-time acceleration with the threshold, once the acceleration exceeds the threshold, it is considered that the device is hit or falls at that time. The threshold can be specified flexibly by those skilled in the art in combination with the actual situation, and the specific value result is not limited in this embodiment.

[0061] It should be noted that the IMU device is a device for measuring the three-axis attitude angle (or angular velocity) and acceleration of an object, which is usually composed of a gyroscope and an accelerometer. Specifically, in the present embodiment, any common lightweight IMU device can be used.

[0062] In S203, the operation processing module outputs a corresponding abnormal state response according to the different types of environmental anomaly detection results and user abnormal behavior detection results, and instructs the AR device to perform corresponding abnormal protection.

[0063] After steps S201 and S202, general detection and identification can be performed for tracking failure caused by different reasons, including lens blocking, device in dark environment or facing large depth of field environment, and user behavior, etc.

[0064] In this step S203, the operation processing module can output different types of abnormal state information for different types of inducing reasons to instruct the execution device to perform corresponding abnormal protection.

[0065] Specifically, when the reason for tracking failure is that the camera faces a large depth of field environment, the AR device can be instructed to perform zero speed update.

[0066] When the reason for tracking failure is that the camera lens is blocked or in a dark environment, the AR device can be instructed to perform 3Dof update.

[0067] When the reason for tracking failure is that the device falls or collides, the AR device can be protected by filtering out an abnormally large acceleration value.

[0068] Through the above steps S201 to S203, compared with the method of not displaying AR content when tracking fails in the related art, in the present embodiment, the operation processing module integrates multi-dimensional detection functions to generally and multi-dimensionally detect the reasons for tracking system failure, and then corresponding abnormal protection can be performed for each abnormal state, thereby greatly improving the user experience.

[0069] In some embodiments, the camera lens blocking detection according to the current image information includes:

[0070] Obtaining the key points of the current image, and obtaining the response intensity and distribution uniformity of the key points;

[0071] Determining whether the current response intensity is in a first preset response intensity interval, and if so, outputting an abnormal state response that the current camera lens is blocked;

[0072] It should be noted that when the camera is in the unblocked state, the response intensity of the image key point is generally larger; and correspondingly, if in the blocked state, the response intensity of the image key point is smaller; in the embodiment, according to the above characteristics, whether the current camera lens is blocked is judged by using the real-time response intensity of the image key point.

[0073] Further, it is judged whether the current distribution uniformity is in the preset uniformity interval, if yes, an abnormal state response is output, that is, the current camera lens is blocked.

[0074] In addition, in the unblocked state, the distribution of the image key point is also relatively uniform; if not in the blocked state, the distribution of the image key point is greatly uneven; in the embodiment, according to this feature, whether the current lens is blocked is judged by using the distribution of the image key point.

[0075] Specifically, a certain numerical interval can be set, such as a first preset response intensity interval reflecting the response intensity of the key point, or a preset uniformity interval reflecting the distribution of the key point; in actual application, the response intensity and the distribution uniformity of the current image are compared with the above two interval values respectively, to judge whether the current distribution uniformity is in the preset uniformity interval, if yes, an abnormal state response is output, that is, the current camera lens is blocked. It should be noted that the above two interval values can also be set by the person skilled in the art in combination with the hardware parameters, and the specific value range is not limited in the embodiment.

[0076] In some embodiments, the same principle as detecting whether the camera lens is blocked can be used to judge whether the current camera is in a dark environment according to the response intensity of the key point in the current image information.

[0077] Specifically, it is judged whether the response intensity of the key point is in a second preset response intensity interval, if yes, an abnormal state response is output, that is, the AR device is currently in a dark environment; wherein the second preset response intensity interval is different from the first preset response intensity interval used for blocking judgment, and the value range thereof should be slightly smaller than the first preset response intensity interval.

[0078] In some embodiments, it is considered that when the camera field of view is directed to a large depth of field environment, it can also cause tracking failure. In the present application, the operation processing module can also detect this type of abnormality, specifically:

[0079] The large depth of field environment monitoring according to the current image information includes: obtaining the length and average disparity of the tracking of the key points of the continuous image frames; according to the length and average disparity, it is judged whether the current camera depth is much larger than the relative displacement of the camera, if yes, an abnormal state response is output, that is, the current camera is directed to a large depth of field environment.

[0080] It should be noted that the shorter the length of the key point tracking between the continuous image frames, the higher the openness of the scene, and the greater the probability of being a large depth of field environment such as the sky, lake surface, etc. The smaller the average disparity between the key points between the continuous image frames, the greater the depth of field of the scene, and the greater the probability of being an environment such as the sky, lake surface, etc. In the present embodiment, the feature is used to determine whether the camera orientation is in a large depth of field environment.

[0081] In some embodiments, for tracking failure caused by user behavior, detection and determination can be performed according to IMU data. Specifically, user abnormal behavior detection according to current IMU information includes:

[0082] According to the current IMU information, the acceleration change amount is obtained, and it is determined whether the acceleration change amount is greater than a first acceleration change threshold. If yes, the abnormal state response is that the AR device is hit or dropped. That is, once the acceleration exceeds a certain threshold, it is considered that the AR glasses have been hit or dropped.

[0083] In some embodiments, the operation processing module determines, according to different types of environment abnormal detection results and user abnormal behavior detection results, that:

[0084] In the case where the abnormal state response is that the current camera lens is blocked, or the AR device is currently in a dark light environment, the AR device is instructed to perform 3DoF update. Specifically, 3DoF update refers to calculating the 3 degrees of freedom attitude of the device based on the gyroscope sensor carried by the device, i.e., rotation around three axes.

[0085] In the case where the abnormal state response is that the current camera orientation is in a large depth of field environment, the AR device is instructed to perform zero speed update. Specifically, when there is no obvious change in the scene image information observed by the camera, the algorithm system will assume that the moving speed of the device where the camera is located is zero, and will support the pose estimation system to make a constrained pose estimation calculation according to the prior information that the speed is zero, thereby improving the stability of the pose estimation in a large depth of field environment.

[0086] In the case where the abnormal state response is that the AR device is hit or dropped, the AR device is instructed to perform abnormal large acceleration value filtering. Specifically, when the AR device is subjected to a violent impact, the accelerometer will measure a very large value, which is not beneficial to the pose estimation system. Therefore, the above-mentioned very large value can be identified and filtered out to avoid inputting these values into the algorithm system.

[0087] Further, after outputting the corresponding abnormal state response and instructing the AR device to perform the corresponding abnormal protection, the method further includes:

[0088] Continuously detect whether the current AR device triggers any one of the following conditions: the camera is blocked, the AR device is currently in a dark light environment, the current camera is facing a large depth of field environment, and if not, switch the pose data to 6Dof.

[0089] Through the above embodiment, a general multi-dimensional tracking anomaly detection method is provided, and for different types of abnormal conditions, appropriate anomaly protection can be performed accordingly, thereby greatly improving the use reliability of the AR device and improving the user experience.

[0090] The embodiment also provides an AR tracking system failure detection system, which is used to implement the above embodiment and preferred embodiment, and will not be described again. As used below, the terms "module", "unit", "sub-unit" and the like can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware, or a combination of software and hardware implementation is also possible and contemplated.

[0091] Figure 3 is a structural block diagram of a tracking system failure detection system according to an embodiment of the present application, as Figure 3 shown, the system is applied to a terminal device, and the image information processing module 30, the IMU information processing module 31 and the response indication module 32, wherein.

[0092] The image processing module 30 is used to obtain current image information, and to perform environment anomaly detection according to the current image information, the environment anomaly detection including: camera lens blocking detection, dark light environment detection and large depth of field environment detection;

[0093] The IMU information processing module 31 is used to obtain current IMU information, and to perform user abnormal behavior anomaly detection according to the current IMU information, including: fall detection and impact detection;

[0094] The response indication module 32 is used to output corresponding abnormal state response according to different types of environment anomaly detection results and user abnormal behavior detection results, and to instruct the AR device to perform corresponding anomaly protection.

[0095] In one embodiment, Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, as Figure 4 shown, an electronic device is provided, which can be a server, and the internal structure diagram thereof can be as Figure 4As shown in the figure. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected by an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capability, the network interface is used to communicate with external terminals through network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement an AR tracking system failure detection method, and the database is used to store data.

[0096] Those skilled in the art can understand that, Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0097] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM) and the like.

[0098] The above-described embodiments only express several implementation manners of the present application, which are described in detail and specifically, but should not be understood as a limitation on the patent scope of the present application. It should be noted that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A method of tracking system failure detection, the method comprising: The application is applied to an AR device, a multi-dimensional abnormality detection function is integrated through an operation processing module, and the method comprises the following steps: Through the operation processing module, current image information is acquired, and environment abnormality detection is performed according to the current image information, wherein the environment abnormality detection comprises camera lens blocking detection, dark light environment detection and large depth of field environment detection; Through the operation processing module, current IMU information is acquired, and user abnormal behavior abnormality detection is performed according to the current IMU information, comprising: fall detection and impact detection; Through the operation processing module, corresponding abnormal state responses are output according to different types of environment abnormality detection results and user abnormal behavior detection results, and the AR device is instructed to perform corresponding abnormal protection, specifically comprising: in the case that the abnormal state response is that the current camera lens is blocked or the AR device is currently in a dark light environment, instructing the AR device to perform 3DoF updating; In the case that the abnormal state response is that the current camera is directed to a large depth of field environment, instructing the AR device to perform zero speed updating; In the case that the abnormal state response is that the AR device is impacted or falls, instructing the AR device to perform abnormal large acceleration value filtering.

2. The method of claim 1, wherein, The camera lens blocking detection according to the current image information comprises the following steps: Key points of a current image are acquired, and response intensity and distribution uniformity of the key points are acquired; It is judged whether the current response intensity is in a first preset response intensity interval, if yes, an abnormal state response is output, that is, the current camera lens is blocked; It is judged whether the current distribution uniformity is in a preset uniformity interval, if yes, an abnormal state response is output, that is, the current camera lens is blocked.

3. The method of claim 1, wherein, The dark light environment detection according to the current image information comprises the following steps: Key points of a current image are acquired, and response intensity of the key points is acquired; It is judged whether the response intensity of the key points is in a second preset response intensity interval, if yes, an abnormal state response is output, that is, the AR device is currently in a dark light environment.

4. The method of claim 1, wherein, The large depth of field environment monitoring according to the current image information comprises the following steps: Length and average disparity of key point tracking of continuous image frames are acquired; According to the length and the average disparity, it is judged whether the current camera depth is much larger than the relative displacement of the camera, if yes, an abnormal state response is output, that is, the current camera is directed to a large depth of field environment.

5. The method of claim 1, wherein, The user abnormal behavior abnormality detection according to the current IMU information comprises the following steps: According to the current IMU information, a current acceleration change amount is acquired, it is judged whether the current acceleration change amount is greater than a first acceleration change threshold, if yes, an abnormal state response is output, that is, the AR device is impacted or falls.

6. The method according to any one of claims 2 to 4, characterized in that, After the corresponding abnormal state responses are output, instructing the AR device to perform corresponding abnormal protection, the method further comprises the following steps: It is continuously detected whether the current AR device triggers any one of the following conditions: the camera is blocked, the AR device is currently in a dark light environment, the current camera is directed to a large depth of field environment, if not, the pose data is switched to 6Dof.

7. A tracking system failure detection system characterized by, The application is applied to an AR device, the system integrates a multi-dimensional abnormality detection function, comprising: an image information processing module, an IMU information processing module and a response instruction module, wherein: The image processing module is configured to acquire current image information, and perform environment anomaly detection according to the current image information, wherein the environment anomaly detection includes camera lens obstruction detection, dark light environment detection, and large depth of field environment detection. The IMU information processing module is configured to acquire current IMU information, and perform user abnormal behavior anomaly detection according to the current IMU information, including fall detection and impact detection. The response indication module is configured to output corresponding abnormal state responses according to different types of environment anomaly detection results and user abnormal behavior detection results, and instruct the AR device to perform corresponding abnormal protection, including: in the case that the abnormal state response is that the current camera lens is obstructed or the AR device is currently in a dark light environment, instructing the AR device to perform 3DoF updating; in the case that the abnormal state response is that the current camera is directed to a large depth of field environment, instructing the AR device to perform zero speed updating; in the case that the abnormal state response is that the AR device is impacted or falls, instructing the AR device to perform abnormal large acceleration value filtering.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 to 6.

Images