Somatosensory-based application program operation method

Abstract

The invention discloses a somatosensory-based application program operation method, apparatus and device, and a computer readable storage medium. The method comprises the steps of obtaining original sensor data of a motion sensor from an intelligent wearable device; performing multi-dimensional coordinate system calibration on the motion sensor to obtain a sensor coordinate system unified matrix subjected to multi-dimensional calibration; performing standardized preprocessing on the original motion data to obtain a standardized sensor data stream; deriving a motion feature vector from the standardized sensor data stream; mapping the motion feature vector into standard input event mapping to obtain a standard input event sequence; obtaining a standardized input event stream capable of cross-platform transmission; and operating a target application program based on the standardized input event data stream. The application program operation method based on somatosensory has the advantages that motion sensor data of the intelligent wearable device are converted into universal input events, and cross-platform man-machine interaction is achieved.

Description

Technical Field

[0001] The present invention relates to the field of somatosensory control technology, and in particular to a somatosensory-based application operation method, device, equipment and computer-readable storage medium. Background Art

[0002] From the emergence of electronic game controllers in the late 1970s and early 1980s, to the innovation of Nintendo's NES controller, to the development of modern multi-function controllers, people's demand for interactive devices has been constantly evolving. Traditional controllers have gradually met the control needs of different game types by integrating technologies such as touch pads, gyroscopes, and motion sensors, but their bulky characteristics have severely limited portability and immediacy, restricting users' interactive experience.

[0003] The rise of smart wearable devices should have brought new possibilities for human-computer interaction, but the reality is disappointing: the vast majority of manufacturers limit these devices to physiological data monitoring, such as steps, heart rate, and blood oxygen, and only a very small number of manufacturers try to apply them to game interaction, and these attempts are often closed and not universal. What is more difficult is that due to the lack of industry standards, there are significant differences in chip models and sensor data processing among manufacturers, resulting in inconsistent data from gyroscopes and accelerometers. The data zero drift problem of low-end devices is particularly serious. These technical barriers further hinder the widespread application of smart wearable devices in the field of interaction.

[0004] In such a context of technological fragmentation and lack of unified standards, how to effectively convert the motion sensor data of smart wearable devices into universal input events and achieve cross-platform and cross-application human-computer interaction has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiment of the present application aims to convert the motion sensor data of the smart wearable device into universal input events by providing a somatosensory-based application operation method, thereby realizing cross-platform and cross-application human-computer interaction.

[0006] To achieve the above objectives, the present application embodiment provides a method for operating an application based on somatosensory, including:

[0007] Establishing a communication connection with a smart wearable device and acquiring raw sensor data of a motion sensor from the smart wearable device;

[0008] According to the original sensor data, the motion sensor is calibrated in a multi-dimensional coordinate system to obtain a sensor coordinate system matrix that has been calibrated in a multi-dimensional manner;

[0009] Performing standardization preprocessing on the original motion data under the sensor coordinate system matrix to obtain a standardized sensor data stream;

[0010] Performing multi-dimensional motion feature extraction on the standardized sensor data stream to obtain a motion feature vector;

[0011] Mapping the motion feature vector into a standard input event mapping to obtain a standard input event sequence, wherein the standard input event includes a keyboard event and a mouse event;

[0012] Performing cross-platform standard interface processing on the standard input event sequence to obtain a standardized input event stream that can be transmitted across platforms;

[0013] Based on the standardized input event data stream, an operation is performed on a target application.

[0014] In one embodiment, the motion sensor is calibrated in a multi-dimensional coordinate system according to the original sensor data to obtain a sensor coordinate system matrix after multi-dimensional calibration, including:

[0015] Based on the original posture data obtained by the motion sensor in multiple preset static postures, the multi-posture coordinate system of the motion sensor is calibrated using the earth's gravity direction reference, and the initial deviation matrix of the motion sensor coordinate system relative to the human anatomical coordinate system is calculated;

[0016] Implement functional dynamic calibration on motion sensors and extract the motion dynamic features within each motion cycle through a preset motion sequence;

[0017] Based on the dynamic characteristics of the motion, a nonlinear error mapping model of the sensor coordinate system is established, and a fitting algorithm is used to generate corresponding coordinate system error correction parameters;

[0018] Verify and evaluate static and dynamic calibration based on the external environment perception module of the terminal device;

[0019] The root mean square error and motion range error of the static calibration results and the dynamic calibration results are calculated to obtain a sensor coordinate system matrix that has been calibrated in multiple dimensions.

[0020] In one embodiment, the external environment perception module includes a camera module;

[0021] Verify and evaluate the static calibration results based on the external environment perception module of the terminal device, including:

[0022] Using the camera module to collect static calibration scene images of the motion sensor worn by the user;

[0023] Preprocessing the static calibration scene image to generate a standardized scene reference image;

[0024] Extract feature points from the standardized scene reference image and perform geometric transformation analysis to obtain an image feature vector;

[0025] Matching and comparing the image feature vector with a static calibration parameter in a motion sensor coordinate system matrix to obtain a static calibration consistency evaluation result;

[0026] Performing a statistical significance test on the static calibration consistency evaluation result to obtain a reliability index of the static calibration result;

[0027] According to the reliability index, the static calibration parameters of the motion sensor coordinate system matrix are adaptively optimized and adjusted to obtain optimized static calibration parameters.

[0028] In one embodiment, the verification and evaluation of the dynamic calibration result is performed according to the external environment perception module of the terminal device, including:

[0029] Performing time-series denoising on the continuous image sequence of the dynamic calibration process collected by the camera module to obtain a smooth dynamic calibration reference image sequence;

[0030] Performing optical flow analysis and motion trajectory reconstruction on the smooth dynamic calibration reference image sequence to obtain a dynamic motion feature map;

[0031] Performing correlation analysis on the dynamic motion feature map and the dynamic calibration parameters of the sensor coordinate system matrix to obtain a dynamic calibration consistency evaluation result;

[0032] Performing multi-scale error decomposition and variance analysis on the dynamic calibration consistency assessment result to obtain error distribution characteristics of the dynamic calibration result;

[0033] Based on the error distribution characteristics of the dynamic calibration result, a nonlinear correction model of the dynamic calibration parameters is constructed to obtain a refined correction value of the dynamic calibration parameters.

[0034] In one embodiment, the raw motion data under a sensor coordinate system matrix is ​​subjected to standardization preprocessing to obtain a standardized sensor data stream, including:

[0035] Perform wavelet transform spectrum decomposition on the original motion data under the sensor coordinate system matrix to obtain multi-scale frequency components;

[0036] Constructing an abnormality recognition model based on machine learning for the multi-scale frequency components, detecting and marking abnormal data segments and bad channels in the signal, and obtaining abnormal signal mapping;

[0037] Applying an adaptive interpolation algorithm to the abnormal data segment in the abnormal signal mapping, reconstructing the signal based on the frequency characteristics of the adjacent normal data segment, and obtaining a corrected frequency component;

[0038] Based on a preset multi-order filter group, the corrected frequency component is subjected to noise filtering to obtain noise reduction processing data;

[0039] Performing an independent component analysis algorithm on the noise reduction processed data to decompose the multidimensional sensor signal into mutually independent functional components to obtain a decoupled signal set;

[0040] Based on the decoupled signal set, a de-noised and reconstructed normalized sensor data stream is generated.

[0041] In one embodiment, performing multi-dimensional motion feature extraction on the standardized sensor data stream to obtain a motion feature vector includes:

[0042] Performing a multi-scale Fourier transform on the standardized sensor data stream to obtain a frequency domain feature representation;

[0043] Performing a time-frequency joint analysis on the frequency domain feature representation to obtain a time-frequency distribution of the motion feature;

[0044] Based on the time-frequency distribution of the motion features, a feature vector clustering algorithm is constructed to obtain a motion feature atomic set;

[0045] Performing semantic relevance analysis on the motion feature atomic set to obtain an action semantic feature map;

[0046] According to the action semantic feature mapping, a multi-dimensional motion feature space vector is generated to obtain a motion feature vector.

[0047] In one embodiment, the motion feature vector is mapped to a standard input event mapping to obtain a standard input event sequence, including:

[0048] Performing semantic analysis and action intention recognition on the motion feature vector to obtain action intention features;

[0049] Based on the action intention features, an action-event mapping rule base is constructed to obtain a mapping conversion model;

[0050] Performing probabilistic semantic matching on the mapping conversion model to obtain a candidate set of input events of action semantics;

[0051] Performing rule filtering and context relevance evaluation on the input event candidate set to obtain a standardized input event subset;

[0052] A standard input event sequence with time sequence correlation is generated according to the standardized input event subset.

[0053] To achieve the above objectives, the present application also provides a somatosensory-based application operating device, including:

[0054] A communication module, used to establish a communication connection with a smart wearable device and obtain raw sensor data of a motion sensor from the smart wearable device;

[0055] A calibration module, used for calibrating the motion sensor in a multi-dimensional coordinate system according to the original sensor data to obtain a sensor coordinate system matrix after multi-dimensional calibration;

[0056] A standardization module, used for performing standardization preprocessing on the original motion data under the sensor coordinate system matrix to obtain a standardized sensor data stream;

[0057] A feature extraction module, used for extracting multi-dimensional motion features from the sensor data stream to obtain a motion feature vector;

[0058] A mapping module, used for mapping the motion feature vector into a standard input event mapping to obtain a standard input event sequence, wherein the standard input event includes a keyboard event and a mouse event;

[0059] An interface processing module, used for performing cross-platform standard interface processing on the standard input event sequence to obtain a standardized input event stream that can be transmitted across platforms;

[0060] The operation module is used to operate the target application based on the standardized input event data stream.

[0061] To achieve the above objectives, an embodiment of the present application also proposes a somatosensory-based application operation device, including a memory, a processor, and a somatosensory-based application operation program stored in the memory and executable on the processor, wherein when the processor executes the somatosensory-based application operation program, it implements the somatosensory-based application operation method as described in any one of the above items.

[0062] To achieve the above objectives, an embodiment of the present application also proposes a computer-readable storage medium, on which a somatosensory-based application operation program is stored. When the somatosensory-based application operation program is executed by a processor, it implements the somatosensory-based application operation method as described in any of the above items.

[0063] The technical solution of this application realizes the precise calibration of motion sensor data through multi-dimensional coordinate system calibration technology, effectively eliminating the systematic error of sensor data between different devices. By combining static posture calibration and functional dynamic calibration, the method can establish a nonlinear error mapping model of the sensor coordinate system, greatly improving the consistency and accuracy of sensor data.

[0064] Secondly, the technical solution of this application adopts multi-level data processing technology, including wavelet transform spectrum decomposition, machine learning anomaly recognition, adaptive interpolation, multi-order filter noise reduction and independent component analysis, to achieve comprehensive processing and reconstruction of raw motion data. This complex signal processing technology significantly improves the quality of sensor data, can effectively filter out noise, remove interference, and decouple multi-dimensional sensor signals into functional components.

[0065] Furthermore, the technical solution of this application builds a set of standardized input event mapping mechanisms, which converts motion data from different smart wearable devices into universal mouse and keyboard events through semantic analysis of motion feature vectors and recognition of action intentions. This cross-platform standardized processing method breaks through the limitations of traditional devices, realizes unified access and standardized processing of heterogeneous sensor devices, and significantly enhances the applicability and versatility of smart wearable devices in the field of human-computer interaction.

[0066] Finally, the technical solution of this application introduces an external environment perception module (such as a camera) to verify and evaluate the calibration results. Through image feature extraction, optical flow analysis, and multi-scale error decomposition, the sensor coordinate system matrix is ​​adaptively optimized and finely corrected. This multimodal fusion verification mechanism not only improves the reliability of the calibration process, but also provides more robust and accurate technical support for complex human-computer interaction scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0068] Figure 1 A module structure diagram of an embodiment of a somatosensory-based application operating device according to the present invention;

[0069] Figure 2 It is a flowchart of an embodiment of a method for operating an application program based on somatosensory according to the present invention;

[0070] Figure 3 FIG. 4 is a flow chart of another embodiment of a method for operating an application program based on somatosensory according to the present invention.

[0071] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0072] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0073] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0074] It should be noted that in the claims, any reference symbols placed between brackets shall not be constructed as limitations to the claims. The presence of "comprising" in the text does not exclude the presence of components or steps not listed in the claims. The quantifier "one" or "an" preceding a component does not exclude the presence of a plurality of such components. The present invention can be implemented by means of hardware comprising several different components and by means of appropriately programmed computers. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of "first", "second", and "third" etc. does not indicate any order, and these words may be interpreted as names.

[0075] like Figure 1 As shown, Figure 1 It is a structural diagram of a server 1 (also called a somatosensory-based application operating device) of a hardware operating environment involved in an embodiment of the present invention.

[0076] The server of the embodiment of the present invention is a device with display function, such as "Internet of Things devices", smart air conditioners, smart lights, smart power supplies with networking functions, AR / VR devices with networking functions, smart speakers, self-driving cars, PCs, smart phones, tablet computers, e-book readers, portable computers, etc.

[0077] like Figure 1 As shown, the server 1 includes: a memory 11, a processor 12 and a network interface 13.

[0078] The memory 11 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the server 1, such as a hard disk of the server 1. In other embodiments, the memory 11 may also be an external storage device of the server 1, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc., equipped on the server 1.

[0079] Furthermore, the memory 11 may also include an internal storage unit of the server 1 and an external storage device. The memory 11 may be used not only to store application software and various data installed on the server 1, such as the code of the somatosensory-based application operating program 10, but also to temporarily store data that has been output or is to be output.

[0080] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 11, such as executing a somatosensory-based application operating program 10.

[0081] The network interface 13 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the server 1 and other electronic devices.

[0082] The network may be the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), and / or a metropolitan area network (MAN). Various devices in the network environment may be configured to connect to the communication network according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and / or Bluetooth (Blue Tooth) communication protocol or a combination thereof.

[0083] Optionally, the server may further include a user interface, which may include a display, an input unit such as a keyboard, and an optional user interface may further include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be referred to as a display screen or a display unit, which is used to display information processed in the server 1 and to display a visual user interface.

[0084] Figure 1 Only the server 1 having components 11-13 and the somatosensory-based application program operating program 10 is shown. It can be understood by those skilled in the art that Figure 1 The structure shown does not constitute a limitation on the server 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0085] In this embodiment, the processor 12 may be used to call the somatosensory-based application operating program stored in the memory 11, and perform the following operations:

[0086] Establishing a communication connection with a smart wearable device and acquiring raw sensor data of a motion sensor from the smart wearable device;

[0087] According to the original sensor data, the motion sensor is calibrated in a multi-dimensional coordinate system to obtain a sensor coordinate system matrix that has been calibrated in a multi-dimensional manner;

[0088] Performing standardization preprocessing on the original motion data under the sensor coordinate system matrix to obtain a standardized sensor data stream;

[0089] Performing multi-dimensional motion feature extraction on the standardized sensor data stream to obtain a motion feature vector;

[0090] Mapping the motion feature vector into a standard input event mapping to obtain a standard input event sequence, wherein the standard input event includes a keyboard event and a mouse event;

[0091] Performing cross-platform standard interface processing on the standard input event sequence to obtain a standardized input event stream that can be transmitted across platforms;

[0092] Based on the standardized input event data stream, an operation is performed on a target application.

[0093] Based on the hardware architecture of the above-mentioned somatosensory-based application operation device, an embodiment of the somatosensory-based application operation method of the present invention is proposed. The somatosensory-based application operation method of the present invention aims to convert the motion sensor data of the smart wearable device into a universal input event to achieve cross-platform and cross-application human-computer interaction.

[0094] Reference Figure 2 , Figure 2 This is an embodiment of the application operation method based on somatosensory of the present invention, and the application operation method based on somatosensory comprises the following steps:

[0095] S10: Establish a communication connection with a smart wearable device, and obtain raw sensor data of a motion sensor from the smart wearable device.

[0096] Specifically, the system first establishes a connection with a smart wearable device (such as a smart bracelet, smart watch or other wearable sensor device) through Bluetooth, Wi-Fi or other wireless communication protocols. After the connection is established, the system obtains raw sensor data from the smart wearable device, which includes acceleration, angular velocity, magnetic field and other information from the motion sensor. These data can reflect the user's motion state and posture changes.

[0097] Optionally, the motion sensor includes but is not limited to an accelerometer, a gyroscope, and a magnetometer.

[0098] S20. Perform multi-dimensional coordinate system calibration on the motion sensor according to the original sensor data to obtain a sensor coordinate system matrix that has undergone multi-dimensional calibration.

[0099] Specifically, the raw sensor data usually has certain errors or deviations, so it needs to be corrected through the calibration process. In this embodiment, the system converts the multi-dimensional coordinates of the sensor through a known calibration method (for example, using a known reference object or calibration scene for data collection) so that the output data of the sensor can accurately reflect the actual motion state. After the multi-dimensional coordinate system calibration, the obtained sensor coordinate system matrix can be used for subsequent data processing and motion analysis.

[0100] S30, performing standardization preprocessing on the original motion data under the sensor coordinate system matrix to obtain a standardized sensor data stream.

[0101] Specifically, the system will perform standardization on the calibrated motion data. Standardization includes noise removal, normalization, filtering and other steps to ensure that the data is compared at a uniform scale. Through standardization, the motion data is converted into a standardized sensor data stream, providing an accurate data basis for subsequent feature extraction and application operations.

[0102] S40, extracting multi-dimensional motion features from the sensor data stream to obtain a motion feature vector.

[0103] Specifically, the sensor data stream contains a large amount of raw data, but in order to perform effective application operations, representative motion features need to be extracted from it. The system analyzes multi-dimensional motion information such as acceleration, rotation angle, motion trajectory, etc., and uses machine learning algorithms or signal processing methods to extract a set of key motion features (for example, motion speed, direction, acceleration change, etc.) to form a motion feature vector. This feature vector can better describe the user's motion behavior.

[0104] S50: Map the motion feature vector to a standard input event mapping to obtain a standard input event sequence.

[0105] Specifically, the system converts motion feature vectors into corresponding standard input events through preset rules or mapping algorithms. For example, a user's gesture can be mapped to a "mouse click" event, and a user's waving action can be mapped to a "keyboard key" event. The system dynamically generates standard input event sequences based on different changes in motion features. These event sequences can simulate user operations on traditional input devices (such as keyboards and mice).

[0106] S60: Perform cross-platform standard interface processing on the standard input event sequence to obtain a standardized input event stream that can be transmitted across platforms.

[0107] Specifically, the standard input event sequence contains multiple events, but these events need to be transmitted and compatible between different platforms. To this end, the system processes the event sequence across platforms and converts it into a standardized input event stream that is compatible with multiple operating systems or application platforms, such as Windows, MacOS, Android, iOS, etc. Through a unified standard interface, it ensures that the input event stream can be correctly identified and executed on different platforms.

[0108] S70. Operate the target application based on the standardized input event data stream.

[0109] Specifically, the system uses the obtained standardized input event stream to simulate the user's operation behavior through traditional input devices (such as a mouse or keyboard) to control the target application. The target application can be any application that supports standard input events, such as word processing software, graphic editing software, game applications, etc. Through somatosensory operation, users can use body movements instead of traditional input methods to operate, achieving a more natural interactive experience.

[0110] It can be understood that the somatosensory-based application operation method of this embodiment obtains the user's motion data through the smart wearable device, and performs multi-dimensional calibration, standardization processing and feature extraction on the data, and then simulates the operation of the traditional input device through the standard input event sequence, achieving the following technical effects:

[0111] 1. This method enables users to operate applications naturally through somatosensory devices, avoiding the limitations of traditional input devices and improving the intuitiveness and immersion of user experience;

[0112] 2. Through cross-platform interface processing, this method can achieve interoperability on multiple operating systems and devices, and has strong applicability;

[0113] 3. While ensuring high-precision motion capture and consistency of input data, this method also effectively simplifies the input operation process and improves the convenience of operation.

[0114] In some embodiments, the motion sensor is calibrated in a multi-dimensional coordinate system according to the raw sensor data to obtain a sensor coordinate system matrix that has been calibrated in a multi-dimensional manner, including:

[0115] S21. Based on the original posture data obtained by the motion sensor in multiple preset static postures, the motion sensor is calibrated in a multi-posture coordinate system using the earth's gravity direction reference, and an initial deviation matrix of the motion sensor coordinate system relative to the human anatomical coordinate system is calculated.

[0116] Specifically, the system first collects raw posture data through motion sensors in a variety of preset static postures (for example, stillness, bending, stretching, and other different postures). These posture data contain information such as acceleration, angular velocity, and magnetic field strength of the sensor in a static state. Based on these static data, the system uses the direction of the earth's gravity (i.e., the vertical gravity acceleration) as a reference to perform multi-posture coordinate system calibration, match the sensor's measurement data with the human anatomical coordinate system, and calculate the initial deviation matrix of the motion sensor coordinate system relative to the human anatomical coordinate system. This deviation matrix is ​​used to correct the sensor output error caused by posture deviation.

[0117] S22, performing functional dynamic calibration on the motion sensor, and extracting the motion dynamic features within each motion cycle through a preset motion sequence.

[0118] Specifically, static calibration can only correct the deviation of the sensor when it is stationary, but during dynamic movement, the output data of the sensor may be affected by more factors. Therefore, the system designs preset action sequences (for example, swinging arms, jumping, turning, etc.) to simulate various dynamic behaviors of users in daily exercise. Through these action sequences, the system extracts the dynamic characteristics of the movement within each action cycle, including information such as acceleration changes, angular velocity changes, and motion trajectory. These features help to further optimize the performance of the sensor in a dynamic state.

[0119] S23. Based on the motion dynamic characteristics, a nonlinear error mapping model of the sensor coordinate system is established, and a fitting algorithm is used to generate corresponding coordinate system error correction parameters.

[0120] Specifically, the system establishes a nonlinear error mapping model of the sensor coordinate system based on the extracted dynamic characteristics of the motion. This model can describe the error distribution and change law of the sensor coordinate system in the dynamic process. Through the fitting algorithm (for example, the least squares method, nonlinear optimization algorithm, etc.), the system can solve the correction parameters to correct the error of the sensor coordinate system in the dynamic state. This step is crucial to improve the accuracy of dynamic motion data.

[0121] S24. Perform verification and evaluation of static calibration and dynamic calibration according to the external environment perception module of the terminal device.

[0122] Specifically, the terminal device is usually equipped with an external environment perception module (such as a camera, laser rangefinder, infrared sensor, radar or environmental sensor, etc.) to obtain external environment information. The system uses these modules to verify and evaluate the static calibration and dynamic calibration results. For example, the terminal device can verify the position and posture of the sensor in static and dynamic processes through a visual recognition algorithm, and compare it with the expected results to evaluate the accuracy and reliability of the calibration process.

[0123] S25. Calculate the root mean square error and the motion range error of the static calibration result and the dynamic calibration result to obtain a sensor coordinate system matrix that has been calibrated in multiple dimensions.

[0124] Specifically, the system calculates the root mean square error (RMSE) of the static and dynamic calibration results and the range of motion error (such as position deviation, angle deviation, etc.), which are used to evaluate the calibration accuracy. Through these calculations, the system can determine whether the calibration results meet the accuracy requirements, and adjust the calibration parameters according to the error results, and finally obtain a sensor coordinate system matrix that has been calibrated in multiple dimensions. This calibration matrix can be used for subsequent sensor data processing and application operations to ensure high accuracy and consistency of sensor data.

[0125] It can be understood that the multi-dimensional coordinate system calibration method in this embodiment effectively eliminates the deviation and error of the motion sensor by combining static and dynamic calibration steps. This method can not only improve the measurement accuracy of the sensor in static conditions, but also correct the nonlinear errors in dynamic motion, thereby improving the accuracy and reliability of the motion data. In addition, the external environment perception module is used for verification and evaluation to further enhance the accuracy and robustness of the calibration process. Through these technical means, this embodiment can provide a high-precision input data stream for somatosensory-based applications, significantly improving the quality of user interaction experience.

[0126] In some embodiments, the external environment perception module includes a camera module.

[0127] Verify and evaluate the static calibration results based on the external environment perception module of the terminal device, including:

[0128] S110: Using a camera module to collect a static calibration scene image of a user wearing a motion sensor.

[0129] Specifically, the terminal device captures static calibration scene images through an integrated camera module when the user wears the motion sensor. The scene usually includes the user's static posture and may contain some calibration tools (such as specific calibration plates, standard coordinate system reference objects, etc.). These images provide a reference for subsequent static calibration verification. The resolution and accuracy of the camera module need to ensure that the details of the user and the calibration tools can be clearly captured to provide sufficient data for subsequent analysis.

[0130] S120: Preprocess the static calibration scene image to generate a standardized scene reference image.

[0131] Specifically, the collected static calibration scene images may be affected by noise, light changes, perspective distortion and other factors, so image preprocessing is required. The system generates standardized scene reference images through image denoising, contrast adjustment, color standardization and other methods. The purpose of this step is to eliminate the influence of external environment and shooting conditions on the image, making the subsequent feature extraction process more stable and accurate.

[0132] S130 , extracting feature points from the standardized scene reference image and performing geometric transformation analysis to obtain an image feature vector.

[0133] Specifically, the system uses computer vision technology (for example, SIFT, SURF, ORB and other feature point detection algorithms) to extract representative feature points from standardized scene reference images. Then, through geometric transformation analysis (such as homography matrix calculation, perspective transformation, etc.), the feature points in the image are analyzed to obtain image feature vectors. These feature vectors contain the spatial position information of each key feature point in the image, providing a basis for subsequent calibration consistency comparison.

[0134] S140, matching and comparing the image feature vector with a static calibration parameter in a motion sensor coordinate system matrix to obtain a static calibration consistency evaluation result.

[0135] Specifically, by matching the image feature vector and the static calibration parameters in the motion sensor coordinate system matrix, the system can compare whether the sensor coordinate system is consistent with the coordinate system in the actual scene image. This matching process is achieved by calculating the relationship between the feature points in the sensor coordinate system and the image coordinate system. Through comparison, the system can evaluate the accuracy and consistency of the static calibration results and obtain the static calibration consistency evaluation results.

[0136] S150: Perform a statistical significance test on the static calibration consistency evaluation result to obtain a reliability index of the static calibration result.

[0137] Specifically, the system uses statistical methods (e.g., t-test, chi-square test, etc.) to perform significance tests on the static calibration consistency evaluation results to determine the reliability of the calibration results. In this way, the system can determine the accuracy of the calibration parameters and avoid result deviations due to random errors or systematic errors. The reliability index reflects the credibility of the static calibration results and provides a basis for subsequent optimization and adjustment.

[0138] S160 . Adaptively optimize and adjust static calibration parameters of a motion sensor coordinate system matrix according to the reliability index to obtain optimized static calibration parameters.

[0139] Specifically, the system uses an adaptive optimization algorithm (such as least squares method, gradient descent method, etc.) to optimize and adjust the static calibration parameters according to the reliability index of the static calibration results. The optimization process aims to improve the calibration accuracy and reduce the deviation caused by sensor errors or external factors. The optimized static calibration parameters will more accurately reflect the actual motion state of the sensor, thereby improving the accuracy of subsequent data processing.

[0140] It can be understood that this embodiment realizes the verification and evaluation of the static calibration results by combining the image analysis of the static calibration scene with the camera module. This method can effectively eliminate the deviation caused by the calibration error, and improve the accuracy and reliability of the static calibration through multi-level analysis and optimization. Specific technical effects include:

[0141] 1. The image preprocessing and feature point extraction technology are used to enhance the stability and accuracy of image data, making the verification of static calibration results more accurate;

[0142] 2. Through statistical significance test, the reliability of static calibration results is ensured and errors caused by accidental factors are avoided;

[0143] 3. The calibration parameters are adjusted through the adaptive optimization algorithm, which further improves the accuracy of static calibration and ensures the accuracy and reliability of subsequent sensor data.

[0144] Therefore, this embodiment effectively improves the calibration accuracy of the motion sensor in the somatosensory operating system, and further enhances the accuracy and fluency of the user interaction experience.

[0145] In some embodiments, the verification and evaluation of the dynamic calibration result is performed according to the external environment perception module of the terminal device, including:

[0146] S210, performing time series denoising processing on the continuous image sequence of the dynamic calibration process collected by the camera module to obtain a smooth dynamic calibration reference image sequence.

[0147] Specifically, the image sequence of the dynamic calibration process collected by the camera module may be affected by factors such as light changes, motion blur, and background noise. Therefore, the system first performs time-series denoising on these continuous image sequences. Denoising can use time-based filtering algorithms (such as Kalman filtering, time-weighted averaging, etc.) to smooth the noise in continuous images and remove interference introduced by motion and environmental factors. Through denoising, the system can obtain a smoother and clearer dynamic calibration reference image sequence to ensure the accuracy of subsequent analysis.

[0148] S220 , performing optical flow analysis and motion trajectory reconstruction on the smooth dynamic calibration reference image sequence to obtain a dynamic motion feature map.

[0149] Specifically, optical flow analysis is a common computer vision technology used to describe the motion of pixels in an image. In this embodiment, the system uses an optical flow algorithm (such as the Lucas-Kanade algorithm) to analyze the smoothed dynamic calibration reference image sequence, and reconstructs the motion trajectory of the object or sensor in the dynamic process by tracking the motion trajectory of each feature point in the image. Through this analysis, the system can obtain a set of dynamic motion feature maps, which contain information such as acceleration, velocity, and displacement of the sensor during the motion process. This information is an important basis for evaluating the dynamic calibration results.

[0150] S230, performing correlation analysis on the dynamic motion feature map and the dynamic calibration parameters of a sensor coordinate system matrix to obtain a dynamic calibration consistency evaluation result.

[0151] Specifically, the system evaluates the match between the dynamic motion feature map and the dynamic calibration parameters in the sensor coordinate system matrix by performing correlation analysis. This process usually uses the Pearson correlation coefficient or other correlation analysis methods to quantify the consistency between the motion features in the image and the sensor data. Through this analysis, the system can determine whether the sensor coordinate system accurately reflects the user's real movement during the dynamic process, thereby obtaining the dynamic calibration consistency evaluation result and evaluating the accuracy of the dynamic calibration parameters.

[0152] S240, performing multi-scale error decomposition and variance analysis on the dynamic calibration consistency evaluation result to obtain error distribution characteristics of the dynamic calibration result.

[0153] Specifically, the system performs multi-scale error decomposition on the results of dynamic calibration consistency assessment and analyzes the distribution of errors at different scales. For example, the system can divide the errors according to time scale, spatial scale or motion stage and analyze them separately. Through variance analysis, the system can obtain the variance and distribution characteristics of errors at different scales, thereby evaluating the error sources and error distribution patterns in the dynamic calibration process. This process helps to identify potential problems in dynamic calibration, such as nonlinear errors, scale inconsistency, etc.

[0154] S250: Based on the error distribution characteristics of the dynamic calibration result, a nonlinear correction model of the dynamic calibration parameters is constructed to obtain a refined correction value of the dynamic calibration parameters.

[0155] Specifically, the system constructs a nonlinear correction model of dynamic calibration parameters based on the error distribution characteristics. The model is used to describe the nonlinear distribution of errors during the dynamic calibration process. Common correction methods include the use of nonlinear optimization algorithms such as high-order polynomial regression and neural networks. By fitting the error distribution, the system can obtain refined correction values ​​for the dynamic calibration parameters. These correction values ​​can effectively reduce calibration errors, improve data accuracy in dynamic processes, and optimize user interaction experience.

[0156] It can be understood that this embodiment realizes the verification and evaluation of the dynamic calibration results by combining the camera module to process and analyze the image sequence of the dynamic calibration process. Specific technical effects include:

[0157] 1. Through time series denoising processing, the noise interference in the image sequence is eliminated and the quality of the dynamic calibration reference image is improved;

[0158] 2. Through optical flow analysis and motion trajectory reconstruction, the motion characteristics of the sensor in the dynamic process are accurately extracted, ensuring the authenticity of the subsequent calibration results;

[0159] 3. Through correlation analysis, error decomposition and variance analysis, the consistency and error distribution of dynamic calibration parameters were comprehensively evaluated, and potential error sources were identified;

[0160] 4. Through the construction of nonlinear correction model, the dynamic calibration parameters are refined, and the accuracy and reliability of dynamic calibration are further improved.

[0161] Therefore, this embodiment can provide a more accurate and stable dynamic calibration data stream for somatosensory-based applications, thereby improving the adaptability and accuracy of the system in complex dynamic environments.

[0162] In some embodiments, the raw motion data under the sensor coordinate system matrix is ​​subjected to standardization preprocessing to obtain a standardized sensor data stream, including:

[0163] S31, performing wavelet transform spectrum decomposition on the original motion data under the sensor coordinate system matrix to obtain multi-scale frequency components.

[0164] Specifically, wavelet transform is a technology widely used in signal processing, which can perform multi-scale spectrum analysis on signals. In this embodiment, the original motion data under the sensor coordinate system matrix is ​​first subjected to wavelet transform, and the signal is separated at different scales by decomposing the frequency components of the signal. Wavelet transform can effectively extract high-frequency and low-frequency components in motion data, which is helpful for subsequent processing and identification of abnormal signals. This process decomposes the original motion data into multiple frequency levels by selecting appropriate wavelet basis functions (such as Haar wavelet, Daubechies wavelet, etc.) to more accurately analyze the time-frequency characteristics of the signal.

[0165] S32: constructing an abnormality recognition model based on machine learning for the multi-scale frequency components, detecting and marking abnormal data segments and bad channels in the signal, and obtaining abnormal signal mapping.

[0166] Specifically, the anomaly recognition model analyzes multi-scale frequency components based on machine learning algorithms (such as support vector machines, decision trees, neural networks, etc.) and identifies abnormal signals by learning the patterns of normal motion data. The model can automatically detect abnormal data segments or bad channels in motion data, such as abnormal signals caused by factors such as equipment failure, environmental interference, or sensor errors. The system builds the model through a training data set and uses it to mark abnormal parts in the actual collected signals and generate abnormal signal mapping. This process can significantly improve data quality and avoid interference of abnormal data on subsequent analysis.

[0167] S33: Apply an adaptive interpolation algorithm to the abnormal data segment in the abnormal signal mapping, perform signal reconstruction based on frequency characteristics of adjacent normal data segments, and obtain a corrected frequency component.

[0168] Specifically, for the identified abnormal data segments, the system uses an adaptive interpolation algorithm to reconstruct them. The interpolation algorithm corrects the abnormal data by using the frequency characteristics of the adjacent normal data segments (such as mean, median, spectral characteristics, etc.). The advantage of adaptive interpolation is that it can automatically adjust the interpolation strategy according to the local characteristics of the data, thereby avoiding over-correction or distortion. Ultimately, the signal reconstructed by this algorithm will be able to more accurately reflect the real motion data of the sensor.

[0169] S34, based on a preset multi-order filter group, filtering out noise on the corrected frequency component to obtain noise reduction processing data.

[0170] Specifically, the corrected frequency components may still contain noise from the environment, equipment, etc. Therefore, the system will use a preset multi-order filter group to filter out the noise. The multi-order filter group is composed of multiple filters with different frequency bandwidths, which can effectively remove noise in different frequency bands. According to the frequency characteristics of the sensor data, the system selects a suitable filter for high-pass, low-pass or band-pass filtering, thereby removing the noise component from the motion signal and retaining the true motion characteristics. Through this step, the quality of the data is further improved.

[0171] S35, executing an independent component analysis algorithm on the noise reduction processed data to decompose the multidimensional sensor signal into mutually independent functional components to obtain a decoupled signal set.

[0172] Specifically, the independent component analysis (ICA) algorithm is a technology commonly used in blind signal separation, which can decompose multidimensional sensor signals into several independent components. In this embodiment, the system uses the ICA algorithm to analyze the multidimensional sensor data after noise reduction processing, and decomposes a set of independent functional signal components. These components may correspond to different motion modes or different sensor functions, such as acceleration, angular velocity, etc. Through ICA, the system can effectively decouple different signal sources, thereby extracting the independent features of each motion mode and improving the accuracy of data processing.

[0173] S36. Based on the decoupled signal set, generate a standardized sensor data stream that has been denoised and reconstructed.

[0174] Specifically, the signal set after ICA decoupling contains independent components of multi-dimensional sensor data, which, after denoising and reconstruction, have fully reflected the user's motion state. The system recombines these decoupled signal sets to generate a standardized sensor data stream. The standardized data stream has been processed by denoising, interpolation, filtering, decoupling, etc., eliminating various interference factors and can truly reflect the user's motion behavior. This standardized data stream can be used by subsequent applications or analysis models for more accurate motion analysis, identification or control.

[0175] It can be understood that this embodiment can significantly improve the accuracy and reliability of motion data by performing multi-level preprocessing on sensor data. Specific technical effects include:

[0176] 1. Perform spectral decomposition of the original motion data through wavelet transform and extract multi-scale frequency components, which helps to more accurately analyze the time-frequency characteristics of motion data;

[0177] 2. Automatically identify abnormal signals and bad channels through anomaly recognition models based on machine learning, thus improving the quality and credibility of data;

[0178] 3. Correct abnormal data segments through adaptive interpolation algorithm to ensure data integrity and consistency;

[0179] 4. Remove noise through multi-order filter groups to further improve the accuracy of the signal;

[0180] 5. Decouple signals through independent component analysis to improve the extraction accuracy of different motion modes.

[0181] Therefore, this embodiment can effectively process and optimize sensor data, provide high-quality, standardized data streams, and meet the needs of complex tasks such as somatosensory applications and motion recognition.

[0182] In some embodiments, performing multi-dimensional motion feature extraction on the standardized sensor data stream to obtain a motion feature vector includes:

[0183] S41, performing a multi-scale Fourier transform on the standardized sensor data stream to obtain a frequency domain feature representation.

[0184] Specifically, multi-scale Fourier transform is a commonly used signal processing technology for converting time domain signals into frequency domain features. In this embodiment, the standardized sensor data stream is Fourier transformed to extract the information of the signal in the frequency domain. By performing multi-scale analysis on the data stream, the motion characteristics of different frequency bands can be identified, and the frequency domain feature representation that is helpful for subsequent analysis can be extracted. This process can reveal the frequency distribution characteristics of motion data and help identify different motion patterns.

[0185] S42, performing a time-frequency joint analysis on the frequency domain feature representation to obtain a time-frequency distribution of the motion feature.

[0186] Specifically, time-frequency joint analysis is an analysis method that combines time and frequency features, which can more comprehensively describe the dynamic changes of signals. In this embodiment, the system combines frequency domain feature representation with time information through time-frequency joint analysis (such as short-time Fourier transform, wavelet transform, etc.) to obtain the time-frequency distribution of motion features. This distribution can reflect the frequency characteristics of motion signals that change over time, provide more detailed information about motion patterns, and help distinguish different motion states.

[0187] S43. Based on the time-frequency distribution of the motion features, a feature vector clustering algorithm is constructed to obtain a motion feature atomic set.

[0188] Specifically, the feature vector clustering algorithm is to find out the potential laws and patterns in the data by classifying the motion features. In this embodiment, the system uses a feature vector clustering algorithm (such as K-means, DBSCAN, etc.) to cluster the motion features based on the time-frequency distribution obtained from the time-frequency joint analysis. The clustering process can classify similar motion features into one category, thereby obtaining motion feature atomic sets, which represent different motion modes or states. The clustered feature atomic sets are helpful for further action recognition and semantic analysis.

[0189] S44, performing semantic relevance analysis on the motion feature atomic set to obtain an action semantic feature map.

[0190] Specifically, semantic relevance analysis aims to explore the potential associations between features and understand their meanings at the semantic level. In this embodiment, the system will perform semantic relevance analysis on the clustered motion feature atomic set to identify the relationship between the feature and the specific action or behavior. By analyzing the semantics of the feature atomic set, the system can map out the features related to the action. For example, some motion features may be related to the "waving" action, while some may be related to the "running" action. This step helps to correspond the motion data with the actual action semantics, thereby improving the accuracy of action recognition.

[0191] S45. Generate a multi-dimensional motion feature space vector according to the action semantic feature mapping to obtain a motion feature vector.

[0192] Specifically, the motion feature vector is a multi-dimensional representation of the motion feature, which can comprehensively describe the dynamic characteristics of the action. In this embodiment, the system generates a multi-dimensional motion feature space vector based on the analysis results obtained from the motion semantic feature mapping. The vector forms a comprehensive motion feature description by fusing the semantic association information of different feature atomic sets. The motion feature vector contains information on the time, frequency, semantics, etc. of the motion, which can effectively represent the user's motion state and be used for subsequent application analysis, motion recognition or control.

[0193] It can be understood that this embodiment can effectively extract valuable motion features from the standardized sensor data stream through the multi-dimensional motion feature extraction method. Specific technical effects include:

[0194] 1. Extract the frequency domain features of motion signals through multi-scale Fourier transform to reveal the frequency distribution of motion data;

[0195] 2. Combine time and frequency information through time-frequency joint analysis to obtain richer motion feature description;

[0196] 3. Extract motion feature atomic sets through feature vector clustering algorithm to identify different motion modes;

[0197] 4. Through semantic relevance analysis, the motion features are matched with the actual actions to obtain the action semantic feature mapping.

[0198] Therefore, this embodiment can efficiently and accurately extract motion features from sensor data, and provide basic data support for various somatosensory applications, motion analysis and control.

[0199] In some embodiments, the motion feature vector is mapped to a standard input event mapping to obtain a standard input event sequence, including:

[0200] S51, performing semantic analysis and action intention recognition on the motion feature vector to obtain action intention features.

[0201] Specifically, the purpose of semantic analysis and action intention recognition is to analyze the user's intention and action intention based on the motion information extracted from the motion feature vector. In this embodiment, the system first performs semantic analysis on the motion feature vector to identify the specific action features contained in the vector. Then, the action intention recognition algorithm (such as a model based on deep learning, a decision tree, etc.) is used to extract action intention features, such as whether the user intends to perform operations such as "clicking", "sliding", and "scrolling". This step associates the motion data with the user's action intention, laying the foundation for subsequent input event mapping.

[0202] S52: Based on the action intention features, construct an action-event mapping rule base to obtain a mapping conversion model.

[0203] Specifically, the action-event mapping rule base is a set of rules that correspond the user's action intentions to specific input events. In this embodiment, the system constructs a mapping rule base based on the action intention features. Each rule in the library defines the relationship between one or more action intentions and corresponding standard input events. Through the rule base, the system can convert the identified action intentions (such as "slide up" or "click") into specific input events (such as mouse clicks, keyboard keys, etc.). The mapping conversion model can efficiently achieve accurate correspondence between actions and input events.

[0204] S53: Perform probabilistic semantic matching on the mapping conversion model to obtain a candidate set of input events of action semantics.

[0205] Specifically, probabilistic semantic matching is to calculate the most likely input event candidate set based on the action intention features and the rules of the mapping rule base. In this embodiment, the system matches the action intention features with the event rule base through a probabilistic model (such as a Bayesian network, a maximum entropy model, etc.) to evaluate the matching probability of each input event rule. In this way, the system can generate one or more input event candidate sets based on the action intention, each candidate set corresponds to one or more input events, and is accompanied by the probability of their occurrence. This process provides multiple alternatives for the selection of subsequent input events.

[0206] S54: Perform rule filtering and context relevance evaluation on the input event candidate set to obtain a standardized input event subset.

[0207] Specifically, rule filtering and context relevance evaluation are the processes of screening the candidate set of input events based on specific context information. In this embodiment, the system filters and evaluates the candidate set of input events based on the current environment, the user's behavior status, or other context information (such as input device type, operation interface status, etc.). For example, some input events may not be applicable in specific situations, or some actions may require specific device support. Through rule filtering and context evaluation, the system can obtain a standardized subset of input events based on the current context, and these event subsets can accurately reflect the user's true operational intentions.

[0208] S55. Generate a standard input event sequence with time sequence correlation according to the standardized input event subset.

[0209] Specifically, the purpose of generating a standard input event sequence is to arrange the input events in chronological order and maintain the temporal correlation between events. In this embodiment, the system generates a temporally correlated input event sequence based on a standardized input event subset. Each input event not only includes the event type (such as a mouse click, a keyboard stroke, etc.), but also includes the timestamp or relative timing information of the event. In this way, the system can ensure the timing of input events, avoid time conflicts or inconsistencies between events, and provide accurate input data streams for subsequent system interactions or application operations.

[0210] It can be understood that this embodiment realizes the effective conversion from the user's body motion to the standard input event through the process of mapping the motion feature vector to the standard input event. The specific technical effects include:

[0211] 1. Through semantic analysis and action intention recognition, the user's action intention is extracted from the motion feature vector, and a basis is provided for the matching of subsequent input events;

[0212] 2. By building an action-event mapping rule base, a direct mapping relationship is established between action intentions and standard input events;

[0213] 3. Through probabilistic semantic matching, multiple event candidate sets are generated according to the occurrence probability of input events to improve the adaptability of the system;

[0214] 4. Through rule filtering and context relevance evaluation, a subset of standard input events that are consistent with the current context is screened out to ensure the accuracy of the events.

[0215] Therefore, this embodiment can efficiently and accurately convert the somatosensory motion data into a standardized input event stream, providing natural and smooth operation control for somatosensory-based applications.

[0216] In some embodiments, a cross-platform standard interface is performed on a standard input event sequence to obtain a standardized input event stream that can be transmitted across platforms, including:

[0217] S61, performing event type normalization processing on the standard input event sequence to obtain a uniformly coded event type standard set.

[0218] Specifically, the purpose of event type normalization is to ensure that the type of input event has a unified representation between different platforms. In this embodiment, the system first normalizes each event in the standard input event sequence to standardize its type and format. For example, different platforms may have different representations for certain input events (such as mouse clicks, keyboard keys, etc.), and through normalization, the system can map the input event types of all platforms to a unified encoding standard, thereby avoiding differences between platforms. This operation converts the types of all input events into a unified coded event type standard set, which is convenient for subsequent processing and conversion.

[0219] S62: Based on the event type standard set, a universal event mapping protocol across operating systems is constructed to obtain a platform-independent event conversion model.

[0220] Specifically, the purpose of constructing a universal event mapping protocol across operating systems is to ensure that input events can be seamlessly transmitted between different operating systems and platforms. In this embodiment, the system builds a platform-independent event conversion model based on a standard set of event types. The model specifies the mapping rules and conversion methods for input events on different platforms (such as Windows, Linux, macOS, iOS, Android, etc.). For example, some operating systems may use different event codes or structures to represent the same input event, and the universal event mapping protocol will ensure that these platforms can correctly understand and convert input events. Through this protocol, the system can ensure the compatibility and correct transmission of input events.

[0221] S63: Perform semantic compatibility analysis on the platform-independent event conversion model to obtain event semantic mapping rules.

[0222] Specifically, the purpose of semantic compatibility analysis is to ensure that the semantics of input events remain consistent on different platforms. In this embodiment, the system performs semantic analysis on the platform-independent event conversion model to evaluate the semantic consistency of input events on different platforms. Since the input devices and interfaces of different platforms may be different, semantic compatibility analysis ensures that the core semantics of the event (such as "click" or "slide") can still be correctly understood and processed even on different hardware and operating systems. Through semantic compatibility analysis, the system can obtain a set of event semantic mapping rules that specify how to map the semantics of input events between different platforms.

[0223] S64. Based on a preset standardized event encapsulation method, the event semantic mapping rule is encoded into a data message to obtain a standardized input event stream data packet.

[0224] Specifically, the purpose of data message encoding is to package the standardized input event stream into a format that can be transmitted on the network. In this embodiment, the system converts the event semantic mapping rules into data messages based on a preset standardized event encapsulation method. Specifically, the system encapsulates each input event and its corresponding semantic features (such as event type, timestamp, coordinate information, etc.) in a certain data format to form a standardized input event stream data packet. This process ensures that input events can be transmitted between different platforms in a unified format while maintaining the integrity and accuracy of the events.

[0225] S65. Encrypt and compress the standardized input event stream data packet to obtain a cross-platform input event stream.

[0226] Specifically, the purpose of encryption and compression processing is to improve the security and efficiency of data transmission. In this embodiment, the system encrypts the standardized input event stream data packet to ensure that the input event data is not tampered with or leaked during transmission. The encryption method can adopt symmetric encryption, asymmetric encryption or hash encryption technology. In addition, the system also compresses the data packet to reduce the bandwidth required for data transmission and improve transmission efficiency. Through encryption and compression processing, the system can generate a cross-platform input event stream that can be efficiently and securely transmitted between multiple operating systems and devices.

[0227] It can be understood that this embodiment realizes efficient transmission of input events between different platforms by processing the standard input event sequence with a cross-platform standard interface. Specific technical effects include:

[0228] 1. Through event type standardization, the system will unify the input event types of different platforms to ensure consistency of cross-platform operations;

[0229] 2. By building a universal event mapping protocol across operating systems, ensure that different platforms can correctly understand and convert input events;

[0230] 3. Through semantic compatibility analysis, the system ensures that input events on different platforms have the same semantic expression, so that events can be correctly parsed;

[0231] 4. Through data message encoding, the system encapsulates the input event stream and prepares for cross-platform transmission;

[0232] 5. Through encryption and compression processing, the security and transmission efficiency of the input event stream are improved, ensuring that the event stream can be transmitted safely and quickly between different platforms.

[0233] Therefore, this embodiment can effectively realize the transmission of cross-platform input event streams based on different platforms, ensure compatibility and operational consistency in a multi-platform environment, and thus enhance the user's cross-platform application experience.

[0234] In addition, refer to Figure 3 The embodiment of the present invention further provides an application operating device based on somatosensory, and the application operating device based on somatosensory includes:

[0235] The communication module 110 is used to establish a communication connection with the smart wearable device and obtain raw sensor data of the motion sensor from the smart wearable device;

[0236] A calibration module 120, configured to perform multi-dimensional coordinate system calibration on the motion sensor according to the original sensor data to obtain a sensor coordinate system matrix after multi-dimensional calibration;

[0237] A standardization module 130, configured to perform standardization preprocessing on the original motion data under the sensor coordinate system matrix to obtain a standardized sensor data stream;

[0238] A feature extraction module 140 is used to extract multi-dimensional motion features from the sensor data stream to obtain a motion feature vector;

[0239] A mapping module 150, configured to map the motion feature vector into a standard input event mapping to obtain a standard input event sequence, wherein the standard input event includes a keyboard event and a mouse event;

[0240] The interface processing module 160 is used to perform cross-platform standard interface processing on the standard input event sequence to obtain a standardized input event stream that can be transmitted across platforms;

[0241] The operation module 170 is used to operate the target application based on the standardized input event data stream.

[0242] The steps implemented by the functional modules of the somatosensory-based application operating device may refer to the various embodiments of the somatosensory-based application operating method of the present invention, and will not be described in detail here.

[0243] In addition, the embodiment of the present invention further proposes a computer-readable storage medium, which may be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, etc. The computer-readable storage medium includes a somatosensory-based application operating program 10. The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the specific implementation of the somatosensory-based application operating method and the server 1, and will not be described in detail herein.

[0244] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0245] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0246] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0247] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0248] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0249] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for operating an application based on somatosensory, characterized in that: include: Establishing a communication connection with a smart wearable device and acquiring raw sensor data of a motion sensor from the smart wearable device; According to the original sensor data, the motion sensor is calibrated in a multi-dimensional coordinate system to obtain a sensor coordinate system matrix that has been calibrated in a multi-dimensional manner; Performing standardization preprocessing on the original motion data under the sensor coordinate system matrix to obtain a standardized sensor data stream; Performing multi-dimensional motion feature extraction on the standardized sensor data stream to obtain a motion feature vector; Mapping the motion feature vector into a standard input event mapping to obtain a standard input event sequence, wherein the standard input event includes a keyboard event and a mouse event; Performing cross-platform standard interface processing on the standard input event sequence to obtain a standardized input event stream that can be transmitted across platforms; Based on the standardized input event data stream, an operation is performed on a target application.

2. The somatosensory-based application operation method according to claim 1, characterized in that: The motion sensor is calibrated in a multi-dimensional coordinate system according to the original sensor data to obtain a sensor coordinate system matrix after multi-dimensional calibration, including: Based on the original posture data obtained by the motion sensor in multiple preset static postures, the multi-posture coordinate system of the motion sensor is calibrated using the earth's gravity direction reference, and the initial deviation matrix of the motion sensor coordinate system relative to the human anatomical coordinate system is calculated; Implement functional dynamic calibration on motion sensors and extract the motion dynamic features within each motion cycle through a preset motion sequence; Based on the dynamic characteristics of the motion, a nonlinear error mapping model of the sensor coordinate system is established, and a fitting algorithm is used to generate corresponding coordinate system error correction parameters; Verify and evaluate static and dynamic calibration based on the external environment perception module of the terminal device; The root mean square error and motion range error of the static calibration results and the dynamic calibration results are calculated to obtain a sensor coordinate system matrix that has been calibrated in multiple dimensions.

3. The somatosensory-based application operation method according to claim 2, characterized in that: The external environment perception module includes a camera module; Verify and evaluate the static calibration results based on the external environment perception module of the terminal device, including: Using the camera module to collect static calibration scene images of the motion sensor worn by the user; Preprocessing the static calibration scene image to generate a standardized scene reference image; Extract feature points from the standardized scene reference image and perform geometric transformation analysis to obtain an image feature vector; Matching and comparing the image feature vector with a static calibration parameter in a motion sensor coordinate system matrix to obtain a static calibration consistency evaluation result; Performing a statistical significance test on the static calibration consistency evaluation result to obtain a reliability index of the static calibration result; According to the reliability index, the static calibration parameters of the motion sensor coordinate system matrix are adaptively optimized and adjusted to obtain optimized static calibration parameters.

4. The method for operating an application program based on somatosensory according to claim 3, characterized in that: Verify and evaluate the dynamic calibration results based on the external environment perception module of the terminal device, including: Performing time-series denoising on the continuous image sequence of the dynamic calibration process collected by the camera module to obtain a smooth dynamic calibration reference image sequence; Performing optical flow analysis and motion trajectory reconstruction on the smooth dynamic calibration reference image sequence to obtain a dynamic motion feature map; Performing correlation analysis on the dynamic motion feature map and the dynamic calibration parameters of the sensor coordinate system matrix to obtain a dynamic calibration consistency evaluation result; Performing multi-scale error decomposition and variance analysis on the dynamic calibration consistency assessment result to obtain error distribution characteristics of the dynamic calibration result; Based on the error distribution characteristics of the dynamic calibration result, a nonlinear correction model of the dynamic calibration parameters is constructed to obtain a refined correction value of the dynamic calibration parameters.

5. The somatosensory-based application operation method according to claim 1, characterized in that: The original motion data under the sensor coordinate system matrix is ​​standardized and preprocessed to obtain a standardized sensor data stream, including: Perform wavelet transform spectrum decomposition on the original motion data under the sensor coordinate system matrix to obtain multi-scale frequency components; Constructing an abnormality recognition model based on machine learning for the multi-scale frequency components, detecting and marking abnormal data segments and bad channels in the signal, and obtaining abnormal signal mapping; Applying an adaptive interpolation algorithm to the abnormal data segment in the abnormal signal mapping, reconstructing the signal based on the frequency characteristics of the adjacent normal data segment, and obtaining a corrected frequency component; Based on a preset multi-order filter group, the corrected frequency component is subjected to noise filtering to obtain noise reduction processing data; Performing an independent component analysis algorithm on the noise reduction processed data to decompose the multidimensional sensor signal into mutually independent functional components to obtain a decoupled signal set; Based on the decoupled signal set, a de-noised and reconstructed normalized sensor data stream is generated.

6. The somatosensory-based application operation method according to claim 1, characterized in that: Performing multi-dimensional motion feature extraction on the standardized sensor data stream to obtain a motion feature vector includes: Performing a multi-scale Fourier transform on the standardized sensor data stream to obtain a frequency domain feature representation; Performing a time-frequency joint analysis on the frequency domain feature representation to obtain a time-frequency distribution of the motion feature; Based on the time-frequency distribution of the motion features, a feature vector clustering algorithm is constructed to obtain a motion feature atomic set; Performing semantic relevance analysis on the motion feature atomic set to obtain an action semantic feature map; According to the action semantic feature mapping, a multi-dimensional motion feature space vector is generated to obtain a motion feature vector.

7. The somatosensory-based application operation method according to claim 1, characterized in that: Map the motion feature vector to a standard input event map to obtain a standard input event sequence, including: Performing semantic analysis and action intention recognition on the motion feature vector to obtain action intention features; Based on the action intention features, an action-event mapping rule base is constructed to obtain a mapping conversion model; Performing probabilistic semantic matching on the mapping conversion model to obtain a candidate set of input events of action semantics; Performing rule filtering and context relevance evaluation on the input event candidate set to obtain a standardized input event subset; A standard input event sequence with time sequence correlation is generated according to the standardized input event subset.

8. A somatosensory-based application operating device, characterized in that: include: A communication module, used to establish a communication connection with a smart wearable device and obtain raw sensor data of a motion sensor from the smart wearable device; A calibration module, used for calibrating the motion sensor in a multi-dimensional coordinate system according to the original sensor data to obtain a sensor coordinate system matrix after multi-dimensional calibration; A standardization module, used for performing standardization preprocessing on the original motion data under the sensor coordinate system matrix to obtain a standardized sensor data stream; A feature extraction module, used for extracting multi-dimensional motion features from the sensor data stream to obtain a motion feature vector; A mapping module, used for mapping the motion feature vector into a standard input event mapping to obtain a standard input event sequence, wherein the standard input event includes a keyboard event and a mouse event; An interface processing module, used for performing cross-platform standard interface processing on the standard input event sequence to obtain a standardized input event stream that can be transmitted across platforms; The operation module is used to operate the target application based on the standardized input event data stream.

9. A somatosensory-based application operating device, characterized in that: The invention comprises a memory, a processor and a somatosensory-based application operating program stored in the memory and executable on the processor, wherein when the processor executes the somatosensory-based application operating program, the somatosensory-based application operating method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a somatosensory-based application operating program, which, when executed by a processor, implements a somatosensory-based application operating method according to any one of claims 1 to 7.

Images