A head motion file preview method and system on an AR glasses device based on IMU

Abstract

The present application discloses a head motion file preview method and system on an AR glasses device based on IMU; the method includes: using an IMU sensor to obtain head motion data of a user wearing an AR glasses device; using a camera device of the AR glasses device to collect facial image data of a user wearing an AR glasses device, and analyzing and obtaining the user's expression data; preprocessing the obtained head motion data and user expression data; using a deep learning algorithm to analyze the head motion data and user expression data, and identify the head motion classification; according to the identified head motion classification and head motion data, completing the data mapping of the movement direction determination and movement distance of the preview file in the AR glasses device, obtaining the running direction and running distance of the preview file, and adjusting the position of the display interface of the preview file on the AR glasses device. The present application can improve the control accuracy and user experience of the AR glasses device when the user browses files.

Description

Technical Field

[0001] The present application relates to the technical field of file preview of AR glasses devices, and in particular to a head-motion file preview method and system on an IMU-based AR glasses device. Background Art

[0002] In recent years, the application of augmented reality (AR) technology in eyewear devices has become increasingly widespread. Users can control the displayed content through simple head movements, which not only improves user convenience, but also greatly enriches the user experience. With the continuous development of AR technology, AR glasses are increasingly used in education, entertainment, office and other fields. How to improve the user experience has become one of the problems that need to be solved in the industry.

[0003] To solve the control problem of AR glasses in file preview and browsing, existing technologies mainly use head movements to control files, including the following methods: one is based on a simple threshold judgment method, which detects head movements and triggers file control when the movement reaches a preset threshold; the other is based on a pattern recognition method, which recognizes different head movements to achieve specific control of files. These two methods have been widely used in practical applications and have become the mainstream means to solve such problems.

[0004] However, the above two methods have certain defects in practical applications. None of the above methods take into account the differences in user habits and the differences in user actions in different states. There are often misjudgments in actual operations, resulting in inaccurate control and failure to meet the accuracy requirements of different users for head action recognition. Therefore, how to improve the control accuracy and user experience of AR glasses when users browse files has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to improve the control accuracy and user experience of AR glasses devices when users browse files, the present application provides a head-motion file preview method and system on an IMU-based AR glasses device.

[0006] In a first aspect, the present application provides a head motion file preview method on an IMU-based AR glasses device, comprising:

[0007] Use the IMU sensor to obtain the head movement data of the user wearing the AR glasses device, including the acceleration and angular velocity of the head movement;

[0008] The camera device of the AR glasses device is used to collect facial image data of the user wearing the AR glasses device, and the facial expression data of the user is obtained by analysis;

[0009] Preprocess the acquired head movement data and user expression data;

[0010] The pre-processed head movement data and user expression data are analyzed using a deep learning algorithm to identify the head movement classification; wherein the head movement classification process includes adaptively and dynamically adjusting the threshold range of each action category to adapt to the user's action preference based on the historical user's head movement data and user expression data;

[0011] According to the recognized head action classification and head movement data, the data mapping of the movement direction and movement distance of the preview file in the AR glasses device is completed according to different preset mapping ratios of different AR glasses device display interface sizes, and the movement direction and movement distance of the preview file are obtained;

[0012] According to the acquired movement direction and movement distance of the preview file, the position of the display interface of the preview file on the AR glasses device is adjusted for the user to preview.

[0013] By adopting the above solution, a dual-modal fusion algorithm integrating head movement data and facial expression data is used to identify head movement classification and adaptively adjust the threshold range to more accurately identify user movements and avoid misjudgment. Data mapping is completed according to different mapping ratios preset for different display interface sizes, and then the display position of the preview file on the AR glasses device is accurately adjusted, thereby improving the user's browsing experience and response speed.

[0014] Preferably, it also includes:

[0015] After adjusting the position of the display interface of the preview file on the AR glasses device, user satisfaction feedback data is collected in real time, and the user satisfaction feedback data includes: user satisfaction feedback data obtained by the user through the AR glasses device button or by collecting the user's hand interaction action and matching the gesture corresponding satisfaction feedback data;

[0016] Compare the collected user satisfaction feedback data with the preset satisfaction level. If it is less than the preset satisfaction level, continue to dynamically adjust the threshold range of each action category until the preset satisfaction level is reached.

[0017] By adopting the above scheme, the threshold range of each head movement category is dynamically adjusted according to the user's satisfaction feedback data until the preset satisfaction is reached, thereby improving the accuracy of head movement control and user satisfaction.

[0018] Preferably, it also includes:

[0019] When adjusting the position of the display interface of the preview file on the AR glasses device according to the acquired movement direction and movement distance of the preview file, determine whether the preview file content displayed on the current display interface has contextual preset associated content. If so, adjust the display ratio of the preview file to display the preview file displayed on the current display interface and the contextual preset associated content with the preview file displayed on the current display interface.

[0020] By adopting the above solution, the context-related content can be automatically judged and displayed to adapt the display interface position, making it more convenient for users to browse files and ensuring the smoothness and continuity of the browsing process.

[0021] Preferably, it also includes:

[0022] Get the application scenario selected by the user currently wearing the AR glasses device;

[0023] According to the currently selected application scenario, a corresponding preset mapping ratio adjustment strategy is matched, and different application scenarios are provided with matching preset mapping ratio adjustment strategies;

[0024] The preset mapping ratio of the display interface size of the AR glasses device is dynamically adjusted by utilizing the mapping ratio adjustment value in the matching preset mapping ratio adjustment strategy.

[0025] By adopting the above solution, the preset mapping ratio adjustment strategy is matched according to the application scenario, and the preset mapping ratio of the display interface size of the AR glasses device is further adjusted, so that the movement direction and distance of the preview file are more in line with the requirements of different application scenarios, thereby improving the user's preview experience in specific scenarios.

[0026] Preferably, it also includes:

[0027] The recognition time of the recognized head movement classification is counted to determine whether the counted head movement classification recognition time is greater than the preset recognition time; if greater, the backup head movement classification recognition model constructed by the deep learning algorithm is used to re-recognize; if not greater, then for each recognized head movement classification, the time for adjusting the position of the preview file on the display interface of the AR glasses device is continued to be counted; if the counted time is continued to be greater than the preset adjustment time and during a preset time period during the use of file preview on the AR glasses device, the frequency of the counted time being greater than the preset adjustment time is greater than the preset frequency, a preview fault prompt message is generated.

[0028] By adopting the above solution, the statistical recognition time and display time are used respectively to complete the action classification recognition in a timely manner, and the failure problems in the preview process are fed back and handled in a timely manner, thereby improving the stability of the system and the user experience.

[0029] Preferably, it also includes:

[0030] Use sensors to collect environmental data about the AR glasses device worn by the user;

[0031] The collected environmental data is input into the IMU sensor sensitivity parameter acquisition model to obtain the optimal IMU sensor sensitivity parameter corresponding to the current environmental data; the IMU sensor sensitivity parameter acquisition model uses a neural network model and is trained and acquired by using historically collected environmental data marked with the optimal IMU sensor sensitivity parameter set by experts;

[0032] Adjust the IMU sensor according to the optimal IMU sensor sensitivity parameters.

[0033] By adopting the above scheme, the optimal IMU sensor sensitivity parameters are dynamically obtained according to environmental data and the IMU sensor is adjusted, thereby improving the adaptability and sensitivity of the IMU sensor in different environments and enhancing the accuracy of head movement recognition.

[0034] Preferably, the preprocessing includes: utilizing time series interpolation technology to perform data frame compensation on the acquired head motion data.

[0035] By adopting the above scheme, the acquired head motion data is compensated for data frames using time series interpolation technology, which improves the continuity and accuracy of the head motion data, thereby improving the smooth transition effect of head motion control.

[0036] In a second aspect, the present application provides a head motion file preview system on an IMU-based AR glasses device, comprising:

[0037] A head motion data acquisition module is used to acquire head motion data of a user wearing an AR glasses device using an IMU sensor, including acceleration and angular velocity of the head motion;

[0038] An expression data acquisition module is used to collect facial image data of a user wearing the AR glasses device using a camera device of the AR glasses device, and analyze and acquire the expression data of the user;

[0039] A data preprocessing module, used for preprocessing the acquired head motion data and user expression data;

[0040] A head movement classification and recognition module is used to analyze the pre-processed head movement data and user expression data using a deep learning algorithm to identify the head movement classification; wherein the head movement classification process includes adaptively and dynamically adjusting the threshold range of each action category to adapt to the user's action preference based on the historical user's head movement data and user expression data;

[0041] The file preview adjustment module is used to complete the data mapping of the movement direction and movement distance of the preview file in the AR glasses device according to the recognized head action classification and head movement data, and the different mapping ratios preset for different AR glasses device display interface sizes, and obtain the movement direction and movement distance of the preview file; according to the obtained movement direction and movement distance of the preview file, adjust the position of the display interface of the preview file on the AR glasses device for user preview.

[0042] By adopting the above solution, the position of the preview file on the AR glasses display interface can be accurately adjusted according to the recognized head movement, thereby improving the consistency and accuracy of the user experience.

[0043] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0044] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a program stored and executable on the memory, wherein the program implements the steps of the above method when executed by the processor.

[0045] In summary, this application has the following beneficial effects:

[0046] 1. Introduce deep learning algorithms to analyze the user's head movement data and expression data, and dynamically adjust the threshold range based on the user's historical data to achieve effective classification of head movements and improve personalized control accuracy; adjust the display position of the preview file on the AR glasses device based on the user's head movement classification, and combine the preset mapping ratios of different interface sizes to achieve a smooth and fluent file preview experience;

[0047] 2. Collect user satisfaction feedback data, and dynamically adjust the threshold range of each head movement category based on the feedback data to improve the accuracy of head movement control and user satisfaction;

[0048] 3. Use IMU sensors to obtain the user's head movement data, and use time series interpolation technology to compensate for the data frames, thereby improving the accuracy and response speed of head movement recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flowchart of a head motion file preview method on an IMU-based AR glasses device described in a specific embodiment;

[0050] Figure 2 It is a structural schematic diagram of the head-motion file preview system on the IMU-based AR glasses device described in a specific embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] like Figure 1 As shown, the embodiment of the present application discloses a head-motion file preview method on an IMU-based AR glasses device, including:

[0053] S1. Collect the user's head movement data.

[0054] Specifically, the IMU sensor is installed on AR glasses to collect head movement data of users wearing AR glasses, including acceleration and angular velocity of head movement.

[0055] Among them, the IMU sensor can use a six-axis combination sensor (such as MPU-6050) or a nine-axis combination sensor (such as BNO055), which is connected to the AR glasses device through a digital interface such as I2C or SPI; the signals collected by the IMU sensor are converted into data packets that capture head movements, which can be implemented by a dedicated chip (such as the STM32 series microcontroller).

[0056] In addition, considering the impact of the environment on the sensitivity of the IMU sensor, sensors are used to collect environmental data of the AR glasses device worn by the user, including: light intensity, temperature and humidity, etc.; the collected environmental data is input into the IMU sensor sensitivity parameter acquisition model to obtain the optimal IMU sensor sensitivity parameters corresponding to the current environmental data; the IMU sensor sensitivity parameter acquisition model uses a neural network model, which is trained and acquired through historically collected environmental data marked with the optimal IMU sensor sensitivity parameters set by experts; the IMU sensor is adjusted according to the optimal IMU sensor sensitivity parameters.

[0057] S2. Collect the user's facial expression data.

[0058] Specifically, a camera device is installed at the front end of the AR glasses to collect facial image data of a user wearing the AR glasses, and an image processing device installed on the AR glasses is used to analyze the collected facial image data to obtain the user's facial expression data.

[0059] The camera device can be of various types, such as an infrared camera (IR Camera) or an RGB camera. The image processing device uses a dedicated image processing chip (such as Hi3519A) to detect the face position in the image and capture the facial area. The Haar feature matching algorithm can be used in conjunction with the AdaBoost classifier to achieve efficient processing of image frames.

[0060] S3. Preprocess the collected head movement data and user expression data.

[0061] Specifically, the preprocessing includes data filtering, data synchronization and standardization, data cleaning and data compensation, etc.

[0062] Among them, a low-pass filter (such as a Butterworth filter) is used to perform signal filtering processing on the collected head movement data and expression data. The data frames collected by the IMU sensor and the camera device are synchronized, and the time synchronization of the head movement data and the expression data is achieved by timestamp matching. Standardization processing is performed on the collected head movement data:

[0063] in, is the head motion data after normalization. The data collected by the original sensor is and The head motion data and expression data are cleaned and compensated, that is, the abnormal data in the head motion data is deleted. In order to ensure the continuity of the head motion data, the time series insertion technology is used to compensate the data frame of the acquired head motion data:

[0064] in, is the head motion data after interpolation. They are and In addition, the data compression unit can compress and store the pre-processed data to reduce storage space.

[0065] S4. Analyze the preprocessed head movement data and user expression data to identify the head movement classification.

[0066] Specifically, machine learning technology is used to identify head movements, and a support vector machine or a deep learning algorithm (such as an LSTM neural network) can be specifically selected for classification. In this embodiment, a deep learning algorithm is selected to analyze the preprocessed head movement data and user expression data to identify the head movement classification.

[0067] Among them, head movements include: shaking the head, nodding, moving up and down, turning the head left and right, etc. The above classifications are pre-set, and each head movement category sets an initial threshold range. These threshold ranges can be set based on historical data, statistical methods or expert experience. When a specific threshold range is reached, it is recognized as belonging to the corresponding head classification.

[0068] A deep learning algorithm is used to perform feature weighting on the preprocessed head motion data and user expression data, extract key features such as the amplitude, speed, duration, facial state, etc. of the movement, and complete the classification of the current head movement based on the threshold range of each movement category dynamically adjusted based on the user's historical head motion data and expression data.

[0069] Among them, considering the head movement preferences of different users, some users shake their heads and nod their heads more frequently, while some users turn left and right with different amplitudes, and the same user shakes and nods their heads more frequently in different states, for example, in an excited state (happy), while in a depressed state (sad), the amplitude of shaking their heads and nodding their heads is relatively gentle. Therefore, the corresponding setting of an adaptive head movement threshold adjustment mechanism can dynamically adjust the action category discrimination threshold of different users or the same user in different states. For a neural network constructed by using a deep learning algorithm, the initial thresholds of various user actions are set, and along with several thresholds for various actions, the historical user head movement data and expression data with labeled action categories are trained, and the thresholds are updated over time, such as:

[0070] in, is the learning rate, A threshold adjustment function calculated based on the action sequence.

[0071] S5. According to the recognized head action classification and head movement data, the running direction of the preview file is determined and mapped to the movement distance.

[0072] Specifically, according to the recognized head action classification and head movement data, the data mapping of the movement direction and movement distance of the preview file in the AR glasses device is completed in accordance with different mapping ratios preset for different AR glasses device display interface sizes.

[0073] Among them, the head movement direction is determined according to the recognized head movement classification and head movement data, and the opposite direction is determined as the movement direction of the preview file; taking into account that different types of AR glasses devices have different display interface sizes, in order to adapt to different display interface sizes, different types of AR glasses devices have different preset mapping ratios for display interface sizes, for example: the first type of AR glasses device display interface size presets a first mapping ratio of 10:1, and the second type of AR glasses device display interface size presets a second mapping ratio of 5:1. Therefore, according to the recognized head movement classification and head movement data, such as the left turn type, the movement data is a moving speed of v and a duration of t, corresponding to a movement of 10 cm to the left. According to the first mapping ratio, the running direction and running distance of the preview file are obtained, which is a movement of 1 cm to the right.

[0074] In addition, in order to improve the user experience, when adjusting the position of the display interface of the preview file on the AR glasses device according to the movement direction and distance of the obtained preview file, it is determined whether there is any contextual preset associated content in the preview file content displayed on the current display interface, and the preset associated content is the content with a contextual association relationship pre-set by the user's preview file; if so, the display ratio of the preview file is adjusted to display the preview file displayed on the current display interface and the contextual preset associated content with the preview file displayed on the current display interface.

[0075] S6. According to the acquired movement direction and movement distance of the preview file, adjust the position of the display interface of the preview file on the AR glasses device for the user to preview.

[0076] Specifically, when the movement direction and distance of the preview file are obtained and the position of the display interface of the preview file on the AR glasses device is adjusted, in order to distinguish the head movement control of the preview file, the moving text content of the preview file is displayed in real time on the display interface with borders of different colors.

[0077] In a specific embodiment, in order to further improve the head motion control accuracy of AR glasses, when the user satisfaction feedback data does not meet expectations, a dynamic adjustment mechanism is started to adjust the action classification threshold; the method also includes:

[0078] After adjusting the position of the preview file on the display interface of the AR glasses device, user satisfaction feedback data is collected in real time.

[0079] Among them, the user satisfaction feedback data includes: user satisfaction feedback data obtained by the user through the button of the AR glasses device, that is, the user uses the button operation of the AR glasses device to select the satisfaction rating option on the display device on the AR glasses device to obtain the user's satisfaction feedback score; or by collecting the user's hand interaction actions and matching the satisfaction feedback data corresponding to the gestures, that is, collecting the user's interaction actions and matching the instructions corresponding to the hand actions, the gesture of two-handed thumbs-up matches the user's very satisfied instruction (100 points), the gesture of one-handed thumbs-up matches the user's satisfied instruction (80 points), and the cross gesture matches the user's dissatisfied instruction (59 points), and the user's satisfaction score is obtained according to the matched instructions.

[0080] Compare the collected user satisfaction feedback data with the preset satisfaction, that is, the user satisfaction score in the user satisfaction feedback data and the preset satisfaction value. If it is less than the preset satisfaction, continue to dynamically adjust the threshold range of each action category, such as: increase training data; until subsequent users' satisfaction with action classification recognition and adjustment of the position of the display interface of the preview file on the AR glasses device reaches the preset satisfaction.

[0081] In a specific embodiment, considering that the same action of the user in different application scenarios may be restricted by the actual site or rules, resulting in different action amplitudes, in order to avoid the influence of the application scenario on the mapping of the head motion data, the method further includes:

[0082] Acquire an application scenario selected by a user currently wearing an AR glasses device; the application scenario is pre-set by the user wearing the AR glasses device when previewing a file wearing the AR glasses device, including typical application scenarios such as outdoor application scenarios and office application scenarios.

[0083] According to the currently selected application scenario, the corresponding preset mapping ratio adjustment strategy is matched, and different application scenarios are set with matching preset mapping ratio adjustment strategies; for example: the outdoor application scenario is relatively wide, allowing the user's head movement to be relatively large, increasing the preview file movement distance corresponding to the head movement, and reducing the preset mapping ratio from 10:1 to 10:2; while the corresponding office application scenario is relatively narrow, allowing the user's head movement to be smaller, reducing the preview file movement distance corresponding to the head movement, and expanding the preset mapping ratio from 10:5 to 10:2.

[0084] The preset mapping ratio of the display interface size of the AR glasses device is dynamically adjusted by utilizing the mapping ratio adjustment value in the matching preset mapping ratio adjustment strategy.

[0085] In addition, from the perspective of application scenarios, in addition to adjusting the mapping ratio accordingly, you can choose to add application scenarios to the training data, so as to train the neural network model constructed using the deep learning algorithm, that is, by setting the initial thresholds for various user actions, and along with several thresholds for various actions, historical user head motion data, expression data and application scenario data with labeled action categories are trained together, and the trained neural network model is used to analyze the current input head motion data, user expression data and application scenarios to obtain the recognized head motion classification.

[0086] In a specific embodiment, considering that the head movement control file preview process may fail to recognize the head movement or adjust the preview file timeout, resulting in failure to meet the user's needs, in order to prevent the user from repeating invalid operations without timely understanding the relevant situation, the method also includes:

[0087] Counting the duration of head movement classification and recognition, and judging whether the counted duration of head movement classification and recognition is greater than a preset duration; the preset duration is set according to the average duration required for successful classification and recognition of various types of movements in history;

[0088] If it is greater than, it indicates that the current head movement classification and recognition has failed, and the backup head movement classification and recognition model constructed by the deep learning algorithm should be used to re-recognize in time. If the recognition still fails after re-recognition using the backup head movement classification and recognition model, a prompt message of recognition failure is generated accordingly;

[0089] If not, then for each identified head movement classification, continue to count the duration of adjusting the position of the display interface of the preview file on the AR glasses device, and determine whether the duration of continued counting is greater than the preset adjustment duration; the preset adjustment duration is set according to the average duration of adjusting the preview file using the movement direction and distance of the historical preview files;

[0090] If the duration of continued statistics is longer than the preset adjustment duration, and during a preset time period (which can be set manually) during the use of file preview on the AR glasses device, the frequency of the continued statistics duration being longer than the preset adjustment duration is greater than the preset frequency, it indicates that there is a fault in the adjustment process of the current preview file according to the head motion data, and a preview fault prompt message is generated.

[0091] like Figure 2 As shown, the embodiment of the present application provides a head-motion file preview system on an AR glasses device based on IMU, which specifically includes:

[0092] The head motion data acquisition module 101 is used to acquire the head motion data of the user wearing the AR glasses device using the IMU sensor, including the acceleration and angular velocity of the head motion;

[0093] The expression data acquisition module 102 is used to collect facial image data of a user wearing the AR glasses device by using a camera device of the AR glasses device, and analyze and acquire the expression data of the user;

[0094] A data preprocessing module 103, used for preprocessing the acquired head motion data and user expression data;

[0095] The head movement classification recognition module 104 is used to analyze the pre-processed head movement data and user expression data using a deep learning algorithm to identify the head movement classification; wherein the head movement classification process includes adaptively and dynamically adjusting the threshold range of each action category to adapt to the user's action preference based on the historical user's head movement data and user expression data;

[0096] The file preview adjustment module 105 is used to complete the data mapping of the movement direction and movement distance of the preview file in the AR glasses device according to the recognized head action classification and head movement data, and the different mapping ratios preset for different AR glasses device display interface sizes, and obtain the movement direction and movement distance of the preview file; according to the obtained movement direction and movement distance of the preview file, adjust the position of the display interface of the preview file on the AR glasses device for user preview.

[0097] In a specific embodiment, the system further includes:

[0098] The file preview feedback module 106 is used to collect user satisfaction feedback data in real time after adjusting the position of the display interface of the preview file on the AR glasses device. The user satisfaction feedback data includes: user satisfaction feedback data obtained by the user through the AR glasses device button or by collecting the user's hand interaction actions and matching the gestures with the corresponding satisfaction feedback data; compare the collected user satisfaction feedback data with the preset satisfaction. If it is less than the preset satisfaction, continue to dynamically adjust the threshold range of each action category until the preset satisfaction is reached.

[0099] The file preview fault prompt module 107 is used to count the recognition time of head movement classification and determine whether the counted head movement classification recognition time is greater than the preset recognition time; if greater, re-recognize using the backup head movement classification recognition model constructed by the deep learning algorithm; if not greater, then for each recognized head movement classification, continue to count the time for adjusting the position of the display interface of the preview file on the AR glasses device; if the counted time is greater than the preset adjustment time and during a preset time period during the use of file preview on the AR glasses device, the frequency of the counted time being greater than the preset adjustment time is greater than the preset frequency, then a preview fault prompt information is generated.

[0100] In a specific embodiment, the head motion data acquisition module 101 in the system is also used to use sensors to collect environmental data of the AR glasses device worn by the user; input the collected environmental data into the IMU sensor sensitivity parameter acquisition model to obtain the optimal IMU sensor sensitivity parameters corresponding to the current environmental data; the IMU sensor sensitivity parameter acquisition model uses a neural network model to train and acquire historically collected environmental data marked with optimal IMU sensor sensitivity parameters set by experts; and adjust the IMU sensor according to the optimal IMU sensor sensitivity parameters.

[0101] In a specific embodiment, the file preview adjustment module 105 in the system is also used to adjust the position of the display interface of the preview file on the AR glasses device according to the acquired movement direction and movement distance of the preview file, and determine whether the preview file content displayed on the current display interface has contextual preset associated content. If so, adjust the display ratio of the preview file to display the preview file displayed on the current display interface and the contextual preset associated content with the preview file displayed on the current display interface.

[0102] In a specific embodiment, the file preview adjustment module 105 in the system is also used to obtain the application scenario selected by the user who is currently wearing the AR glasses device; match the corresponding preset mapping ratio adjustment strategy according to the currently selected application scenario, and different application scenarios are provided with matching preset mapping ratio adjustment strategies; use the mapping ratio adjustment value in the matching preset mapping ratio adjustment strategy to dynamically adjust the preset mapping ratio of the display interface size of the AR glasses device.

[0103] The embodiment of the present application also discloses a computer-readable storage medium.

[0104] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executes a head-motion file preview method on the IMU-based AR glasses device as described above. The computer-readable storage medium includes, for example: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0105] The embodiment of the present application also discloses a computer device.

[0106] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and executes the above-mentioned head motion file preview method on the IMU-based AR glasses device.

[0107] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A head motion file preview method on an IMU-based AR glasses device, characterized in that: include: Use the IMU sensor to obtain the head movement data of the user wearing the AR glasses device, including the acceleration and angular velocity of the head movement; The camera device of the AR glasses device is used to collect facial image data of the user wearing the AR glasses device, and the facial expression data of the user is obtained by analysis; Preprocessing is performed on the acquired head motion data and user expression data; the preprocessing includes data filtering, data synchronization and standardization, data cleaning and data compensation; a low-pass filter is used to perform signal filtering processing on the collected head motion data and expression data; Synchronize the data frames collected by the IMU sensor and the camera device, and use timestamp matching to achieve time synchronization between head movement data and expression data; perform standardized processing on the collected head movement data: ;in, is the head motion data after normalization. The data collected by the original sensor is and The mean and standard deviation are respectively; data cleaning and data compensation are performed on the head motion data and expression data, including: deleting abnormal data in the head motion data, using time series pinning technology to perform data frame compensation on the acquired head motion data: ;in, is the head motion data after interpolation. They are and Head movement data at all times; The pre-processed head movement data and user expression data are analyzed using a deep learning algorithm to identify the head movement classification; wherein the head movement classification process includes adaptively and dynamically adjusting the threshold range of each action category to adapt to the user's action preference based on the historical user's head movement data and user expression data; According to the recognized head action classification and head movement data, the data mapping of the movement direction and movement distance of the preview file in the AR glasses device is completed in accordance with different preset mapping ratios of different AR glasses device display interface sizes, and the movement direction and movement distance of the preview file are obtained; According to the obtained movement direction and movement distance of the preview file, the position of the display interface of the preview file on the AR glasses device is adjusted for the user to preview.

2. The head motion file preview method on the IMU-based AR glasses device according to claim 1, characterized in that: Also includes: After adjusting the position of the display interface of the preview file on the AR glasses device, user satisfaction feedback data is collected in real time, and the user satisfaction feedback data includes: user satisfaction feedback data obtained by the user through the AR glasses device button or by collecting the user's hand interaction action and matching the gesture corresponding satisfaction feedback data; Compare the collected user satisfaction feedback data with the preset satisfaction level. If it is less than the preset satisfaction level, continue to dynamically adjust the threshold range of each action category until the preset satisfaction level is reached.

3. The head motion file preview method on the IMU-based AR glasses device according to claim 1, characterized in that: Also includes: When adjusting the position of the display interface of the preview file on the AR glasses device according to the acquired movement direction and movement distance of the preview file, determining whether the preview file content displayed on the current display interface has contextual preset associated content; If so, the display ratio of the preview file is adjusted to display the preview file displayed on the current display interface and the content associated with the preview file displayed on the current display interface in a context preset manner.

4. The head motion file preview method on the IMU-based AR glasses device according to claim 1, characterized in that: Also includes: Get the application scenario selected by the user currently wearing the AR glasses device; According to the currently selected application scenario, a corresponding preset mapping ratio adjustment strategy is matched, and different application scenarios are set with matching preset mapping ratio adjustment strategies; The preset mapping ratio of the display interface size of the AR glasses device is dynamically adjusted by utilizing the mapping ratio adjustment value in the matching preset mapping ratio adjustment strategy.

5. The head motion file preview method on the IMU-based AR glasses device according to claim 1, characterized in that: Also includes: Counting the duration of head movement classification recognition, and determining whether the counted duration of head movement classification recognition is greater than a preset recognition duration; If it is greater, the backup head action classification and recognition model constructed by the deep learning algorithm is used to re-recognize; If it is not greater than, then for each recognized head movement classification, continue to count the duration of the corresponding adjustment of the position of the preview file on the display interface of the AR glasses device; if the continued counting time is greater than the preset adjustment time and during a preset time period during the use of file preview on the AR glasses device, the frequency of the continued counting time greater than the preset adjustment time is greater than the preset frequency, then a preview fault prompt message is generated.

6. The head motion file preview method on the IMU-based AR glasses device according to claim 1, characterized in that: Also includes: Use sensors to collect environmental data about the AR glasses device worn by the user; The collected environmental data is input into the IMU sensor sensitivity parameter acquisition model to obtain the optimal IMU sensor sensitivity parameter corresponding to the current environmental data; the IMU sensor sensitivity parameter acquisition model uses a neural network model and is trained and acquired by using historically collected environmental data marked with the optimal IMU sensor sensitivity parameter set by experts; Adjust the IMU sensor according to the optimal IMU sensor sensitivity parameters.

7. The head motion file preview method on the IMU-based AR glasses device according to claim 1, characterized in that: The preprocessing includes: using time series frame insertion technology to perform data frame compensation on the acquired head motion data.

8. A head-motion file preview system on an IMU-based AR glasses device, characterized in that: include: A head motion data acquisition module is used to acquire head motion data of a user wearing an AR glasses device using an IMU sensor, including acceleration and angular velocity of the head motion; An expression data acquisition module is used to collect facial image data of a user wearing the AR glasses device using a camera device of the AR glasses device, and analyze and acquire the expression data of the user; A data preprocessing module is used to preprocess the acquired head motion data and user expression data; the preprocessing includes data filtering, data synchronization and standardization, data cleaning and data compensation; a low-pass filter is used to perform signal filtering processing on the collected head motion data and expression data; Synchronize the data frames collected by the IMU sensor and the camera device, and use timestamp matching to achieve time synchronization between head movement data and expression data; perform standardized processing on the collected head movement data: ;in, is the head motion data after normalization. The data collected by the original sensor is and The mean and standard deviation are respectively; data cleaning and data compensation are performed on the head motion data and expression data, including: deleting abnormal data in the head motion data, using time series pinning technology to perform data frame compensation on the acquired head motion data: ;in, is the head motion data after interpolation. They are and Head movement data at all times; A head movement classification and recognition module is used to analyze the pre-processed head movement data and user expression data using a deep learning algorithm to identify the head movement classification; wherein the head movement classification process includes adaptively and dynamically adjusting the threshold range of each action category to adapt to the user's action preference based on the historical user's head movement data and user expression data; The file preview adjustment module is used to complete the data mapping of the movement direction and movement distance of the preview file in the AR glasses device according to the recognized head action classification and head movement data, and the different mapping ratios preset for different AR glasses device display interface sizes, and obtain the movement direction and movement distance of the preview file; according to the obtained movement direction and movement distance of the preview file, adjust the position of the display interface of the preview file on the AR glasses device for user preview.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises a memory, a processor and a program stored and executable on the memory, and the program implements the steps of the method according to any one of claims 1 to 7 when executed by the processor.

Images