A continuous in-air handwriting stroke recognition method and system for extended reality

Abstract

A method and system for continuous aerial handwriting stroke recognition in extended reality includes: sampling over time to acquire the three-dimensional position sequence of key finger points during user writing and the 6-DOF pose information of an XR computing device at the sampling time; constructing and training a lightweight neural network model, the model flow of which is as follows: preprocessing the input data and constructing kinematic features, then passing the kinematic features through a deep separable temporal convolutional network to output the writing probability and boundary probability at each sampling time, with the boundary probability used for auxiliary supervision during training; passing the sampled data through the trained lightweight neural network model to infer and output the writing probability at the sampling time, and extracting connected segments according to a threshold to recover the stroke segment set, thereby achieving visual interaction with the user. This invention relates to the field of information technology and can automatically identify the position of writing strokes and their boundaries from continuous hand movements without relying on virtual plane constraints or excessive reliance on explicit switching actions.

Description

Technical Field

[0001] This invention relates to a method and system for continuous aerial handwriting recognition in extended reality, and relates to the field of information technology. Background Technology

[0002] Extended Reality (XR) systems are gradually becoming important platforms for education and training, industrial design, remote collaboration, and content creation. Unlike two-dimensional desktop environments, XR interaction and information presentation are completed in three-dimensional space, where users often need to annotate text, make spatial comments, or collaborate on writing near objects. While keyboard and voice input methods are efficient, they tend to be more "symbolic input," making it difficult to naturally preserve individual writing styles, spatial layouts, and trajectory expressions. Therefore, in scenarios such as immersive teaching, spatial whiteboard collaboration, rapid sketching annotation, free-form spatial annotation, and digital signatures, air handwriting still has irreplaceable value.

[0003] However, aerial handwriting differs fundamentally from traditional two-dimensional pen and paper writing. Traditional two-dimensional writing relies on a physical paper surface or screen as a reference; the act of putting down the pen and lifting it can be naturally distinguished by the contact state, and the movement of lifting the pen is usually not rendered as a stroke. The writer can also stably control the micro-movements of the hand with the help of friction and tactile feedback. In contrast, in a three-dimensional aerial environment, the trajectory of the finger / hand is a continuously sampled temporal signal. The trajectory inevitably contains a large number of movements "from one stroke to the next," word / line breaks, pauses and adjustments, and relative displacements caused by line of sight / head movements. If the system cannot stably distinguish between written strokes and non-written movements at the motion level, the rendering results will have problems such as incorrect stroke connections, broken strokes, boundary jitter, and stroke drift, thereby reducing readability and editability, and affecting subsequent standard character recognition or input method recognition modules.

[0004] In existing technologies, common schemes for marking writing states in the air can be roughly categorized into three types. The first type is distance-triggered schemes based on virtual planes or physical agents: triggering "pen strokes" by a distance threshold between the finger and a virtual plane, or by contact / approach with a physical object. The problem is that without tactile feedback, users need to continuously pay attention to depth and distance while maintaining posture, easily leading to additional cognitive burden and muscle fatigue. Furthermore, depth control stability is poor across different users and scenarios. The second type is explicit on / off schemes: switching writing states via buttons, handle triggers, pinch / release, or specific gestures. This scheme breaks continuous writing into frequent "command actions," significantly increasing the number of triggers in high-stroke-frequency writing such as Chinese, disrupting writing fluency and causing finger fatigue. The third type is rule-based threshold schemes: using heuristic rules such as speed thresholds, pause thresholds, and angle thresholds to determine writing / non-writing states. However, significant differences exist between different user writing habits, device noise, sampling rates, and task content, making it difficult to uniformly set thresholds, resulting in insufficient generalization and potential boundary jitter during real-time operation.

[0005] On the other hand, although machine learning methods can be used to classify continuous trajectories point by point, the cost of manually annotating continuous input data frame by frame is extremely high, and the "stroke boundary" itself has an acceptable temporal ambiguity range in terms of visual effect: the offset of the boundary several frames before and after does not necessarily affect the readability of the stroke. If hard boundary labels are directly used for supervision, the model is prone to overfitting to specific acquisition methods or specific user action patterns, resulting in unstable output and boundary oscillations during real-time inference.

[0006] Therefore, how to automatically identify the position of writing strokes and their boundaries from continuous hand movements without relying on virtual plane constraints or excessively relying on explicit switching actions has become a key technical issue of concern to technicians. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a method and system for continuous aerial handwriting stroke recognition in extended reality, which can automatically identify the position of writing strokes and their boundaries from continuous hand movements without relying on virtual plane constraints or excessively relying on explicit switching actions.

[0008] To achieve the above objectives, the present invention provides a method for continuous aerial handwritten stroke recognition in extended reality, comprising:

[0009] Step 1: Sampling by time to obtain the three-dimensional position sequence of key finger points when the user writes, and the 6-DOF pose information of the XR computing device at the corresponding sampling time;

[0010] Step 2: Construct and train a lightweight neural network model. The workflow of the lightweight neural network model is as follows: preprocess the input data and construct the corresponding kinematic features. Then, pass the constructed kinematic features through a deep separable temporal convolutional network to output the writing probability and boundary probability at each sampling time. The boundary probability is used for auxiliary supervision during training.

[0011] Step 3: The sampled data is processed through a trained lightweight neural network model to infer and output the writing probability at the sampling time. Connected segments are extracted based on the threshold to recover the stroke segment set, thereby realizing visual interaction with the user.

[0012] To achieve the above objectives, the present invention also provides a continuous air handwriting recognition system for extended reality, comprising:

[0013] The data acquisition device samples over time to obtain the three-dimensional position sequence of key finger points when the user writes, as well as the 6-DOF pose information of the XR computing device at the corresponding sampling time.

[0014] The neural network model device constructs and trains a lightweight neural network model. The workflow of the lightweight neural network model is as follows: the input data is preprocessed and corresponding kinematic features are constructed. Then, the constructed kinematic features are passed through a deep separable temporal convolutional network to output the writing probability and boundary probability at each sampling time. The boundary probability is used for auxiliary supervision during training.

[0015] The inference application device uses a trained lightweight neural network model to infer the writing probability at the sampling time from the sampled data, and extracts connected segments based on a threshold to recover the stroke segment set, thereby realizing visual interaction with the user.

[0016] To achieve the above objectives, the present invention also provides a computing device, comprising:

[0017] Memory and processor;

[0018] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the continuous air handwriting stroke recognition method for extended reality.

[0019] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the continuous air handwriting recognition method for extended reality.

[0020] Compared with existing technologies, the beneficial effects of this invention are: This invention supports real-time inference and stable rendering on the device side, and obtains high-quality supervised data in a low-cost manner, while possessing cross-user generalization capabilities and rapid personalization capabilities for new users; this invention also proposes a low-cost self-annotation training data acquisition mechanism for XR air handwriting: it automatically generates time-by-time writing / non-writing labels using the user-familiar and easily learned pinch-release interaction state, thereby avoiding the high cost of manual annotation of continuous trajectories frame by frame, and making the acquired data similar to natural handwriting in kinematic features, which can be used for basic model training and rapid personalization for new users; this invention also proposes a sliding window fixed extension for real-time interaction on the device side. The delayed inference framework introduces historical and future contexts in each inference step, achieving more stable boundary localization through "delayed confirmation," and combining this with an output queue to achieve continuous and smooth stroke segment output, thereby significantly reducing boundary jitter, incorrect stroke connections, and broken strokes within a controllable delay range. This invention also proposes a lightweight dual-branch temporal network structure and a boundary-aware training strategy: using a Deeply Separable Temporal Convolutional Network (DS-TCN) as the backbone to meet the computational constraints of the XR endpoint, introducing a boundary branch in addition to the classification branch, and enhancing the learning of state transition positions through soft boundary supervision and branch alignment loss, making the model more robust to boundary temporal ambiguity and sensor noise, and outputting more coherent and stable stroke segments. The invention also proposes a training mechanism that balances cross-user generalization and rapid personalization: treating "user / scenario" as the task domain, using self-labeled data for rapid adaptation within a meta-learning framework, and combining it with a small amount of manually corrected continuous writing data for cross-domain generalization updates, thus providing basic usability during cold starts for new users and enabling rapid improvement with a small number of self-labeled samples; the invention also proposes an asynchronous visualization closed-loop interaction scheme for fixed-delay inference: during model inference, a semi-transparent trajectory guide line is first displayed as immediate feedback, and after the recognition result is reached, the guide line is covered and hidden with stable strokes, thereby maintaining recognition stability while preserving continuous interactive perceptibility and system trust; at the interaction level... This invention transfers "writing state control" from the user to the system through automatic stroke recognition. Users can move their hands continuously as if writing naturally without having to perform additional command actions for each stroke. This reduces the attention switching and motion burden caused by virtual plane distance control or frequent gesture switching, improves writing fluency, and reduces the risk of finger and arm fatigue. At the data and deployment level, this invention utilizes a pinch-release gesture self-annotation acquisition mechanism to significantly reduce the cost of training data acquisition. This enables the system to collect usable supervisory data for new users in a short time and complete rapid personalization without the need for complex annotation tools. This solution is conducive to expansion to multi-user, multi-language writing systems and multi-device sampling conditions.At the algorithm and real-time level, this invention employs a lightweight temporal network combined with sliding window fixed-delay inference, enabling the system to achieve low-latency inference and stable output even under resource-constrained conditions on the XR edge. Through boundary branching, soft boundary supervision, and branch alignment training strategies, it effectively reduces broken strokes, incorrect stroke connections, and boundary jitter, while improving robustness to noise and boundary blur. At the generalization and maintenance level, this invention optionally introduces meta-learning and cross-domain generalization training, enabling the model to have basic usable zero-sample capabilities for unseen users and to achieve rapid adaptation with a small number of self-labeled samples, thereby reducing the deployment and maintenance costs of the model in multi-user environments and improving the long-term availability of the system. At the user experience level, the asynchronous visualization and output queue mechanism of this invention provides instant feedback and smooth presentation for fixed-delay inference, allowing users to perceive the writing trajectory without waiting for the model output and obtain a stable and readable stroke presentation after the result arrives, thereby improving system trust and usability. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present invention of a method for recognizing continuous aerial handwritten strokes in extended reality.

[0022] Figure 2 This is a schematic diagram illustrating a process of reasoning through a sliding window and displaying the result using an asynchronous visualization strategy, as shown in an exemplary embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram illustrating the structure of a continuous aerial handwriting recognition system for extended reality, as shown in an exemplary embodiment of the present invention.

[0024] Figure 4 This is a comparison chart of offline quantitative recognition results of different stroke recognition methods obtained through experimental verification in an XR scenario, as shown in an exemplary embodiment of the present invention.

[0025] Figure 5 This is a comparative experimental evaluation result diagram showing the results obtained through experimental verification in an XR scenario, as illustrated in an exemplary embodiment of the present invention.

[0026] Figure 6 This is a diagram illustrating the NASA-TLX scale results of an online user study obtained through experimental verification in an XR scenario, as shown in an exemplary embodiment of the present invention.

[0027] Figure 7 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0029] This invention proposes a continuous air handwritten stroke recognition method and system for XR: First, supervised data is obtained in a low-cost manner to support cross-user modeling; second, a lightweight temporal model is constructed with kinematic features as the core; then, cross-domain meta-learning is used to improve cross-user generalization and support rapid personalization; finally, stable output is achieved on the device side with fixed-delay inference using a sliding window, and asynchronous visualization is used to compensate for the delay perception, forming a usable real-time interactive system.

[0030] like Figure 1 As shown, the present invention provides a method for continuous aerial handwritten stroke recognition in extended reality, comprising:

[0031] Step 1: Sampling by time to obtain the 3D position sequence of the user's finger key points during writing, and the 6-DOF pose information of the XR computing device at the corresponding sampling time. The 3D position sequence of the finger key points is represented as follows: , Indicates time Key points of the fingers, This refers to the number of sampling times; the XR computing device operates at the sampling time. The 6-DOF pose information can be obtained from the rotation matrix. With translation vector Characterization;

[0032] This invention can further specify the time. Key points of the fingers The average point was set at the fingertips of the thumb and index finger to improve trajectory stability under occlusion and shaking conditions and reduce the impact of individual differences on the sampling point.

[0033] Step 2: Construct and train a lightweight neural network model. The workflow of the lightweight neural network model is as follows: preprocess the input data and construct the corresponding kinematic features. Then, pass the constructed kinematic features through a deep separable temporal convolutional network to output the writing probability and boundary probability at each sampling time. The boundary probability is used for auxiliary supervision during training.

[0034] Step 3: The sampled data is processed through a trained lightweight neural network model to infer and output the writing probability at the sampling time. Connected segments are extracted based on the threshold to recover the stroke segment set, which is used for subsequent real-time rendering, stroke-level editing and standard character recognition, thereby realizing visual interaction with the user.

[0035] The core difficulty of continuous air handwriting lies in the lack of natural "pen-on / pen-off" contact signals, making it impossible to annotate the continuous trajectory frame by frame at low cost. Since the trajectory acquired by the pinch-release gesture has a strong similarity in kinematic patterns to natural handwriting, it can provide effective samples for the model to learn the "writing / non-writing" discrimination boundary and avoid expensive frame-by-frame manual annotation. To obtain supervised data in a low-cost manner to support cross-user modeling, this invention can also collect self-annotated training data, including:

[0036] The pinch-release gesture interaction state is used as a supervision signal to collect self-labeled training data. That is, when the gesture is in the pinch state, the collected key point positions are labeled as the writing state (i.e.,...). When the gesture is in the release state, the collected key point positions are marked as non-writing movement (i.e., In this way, the model can use pinch-release gestures to obtain supervision during the training / personalization phase, while allowing users to input with more natural continuous movements during the inference phase (only one activation gesture is needed to enable recognition).

[0037] The core recognition capability of this invention is implemented by a lightweight neural network model. The model performs temporal modeling of continuous trajectories on the edge and simultaneously outputs the writing probability and boundary probability. To improve consistency across users and scenarios, the model input undergoes necessary preprocessing and feature construction before entering the network; to ensure real-time performance on the edge, the network structure adopts a lightweight temporal backbone. The specific workflow of the lightweight neural network model is as follows:

[0038] Step A1, Preprocessing: Based on the camera pose of the center frame of the window, transform the key point position of each moment in the three-dimensional position sequence of the finger key points from the world coordinate system to the camera coordinate system of the XR computing device, and normalize the scale within the window according to the maximum span of the trajectory points within the window.

[0039] In terms of preprocessing, in order to reduce the impact of differences in user standing position, head display posture, writing position and world coordinate system, this invention transforms the trajectory points from the world coordinate system to the XR computing device camera coordinate system according to the camera pose of the center frame of the window in each inference window, and performs window scale normalization according to the maximum span of the trajectory points in the window, so that the model focuses on "relative shape and motion change" rather than absolute position.

[0040] Step A2: Constructing kinematic features: Convert the keypoint positions at each moment in the 3D position sequence of finger keypoints into kinematic feature vectors. , The kinematic feature vector represents time t. The kinematic features include the position of the trajectory point and its changes over time, specifically including velocity, tangential acceleration, and normal acceleration.

[0041] In terms of features, this invention does not rely on specific text content, but uses kinematic features to characterize "written / non-written" and forms a kinematic feature vector at each sampling time. As the input to the model below, this kinematic representation is independent of the text content, and therefore can provide stable and consistent discrimination cues under different writing systems.

[0042] Step A3: Input the kinematic feature vectors into a depthwise separable temporal convolutional network. This network consists of several stacked depthwise separable temporal convolutional modules. Each module contains at least one-dimensional depthwise convolution, non-linear activation, pointwise convolution, and residual connections. This is used to model temporal dependencies from local to longer time domains with low computational cost. The network employs a two-branch output: a classification branch (classification head) and an auxiliary boundary branch (boundary head). The classification branch outputs the writing probability at each time step. It is used to determine whether each point belongs to the stroke of the pen; the auxiliary boundary branch outputs the state transition boundary probability at each time step. This is used to explicitly enhance the localization of stroke start and end boundaries. During training, a two-branch output is used for concatenation and supervision, while during inference, only the results of the classification branch are used, thus effectively meeting the requirements of "speed, coherence, and boundary stability" for continuous air handwriting.

[0043] In step two, the training strategy for the lightweight neural network model consists of two parts: First, addressing the objective fact that "boundaries are inherently ambiguous," boundary-aware supervision is used to improve boundary robustness and suppress online jitter; second, addressing "user differences and data domain differences," cross-domain meta-learning is employed to improve cross-user generalization and support rapid personalization. Specifically:

[0044] (1) Boundary-aware supervision, the specific implementation process is as follows:

[0045] Step B1: Obtain point-by-point hard labels: , This represents the label at sampling time t. Indicates the writing state (pen-down, writing). To represent non-writing movement (pen-up), the state transition moment of the label is set as the boundary center, and all boundary centers form a boundary center set. Then, a soft boundary label is constructed within a preset area of ​​each boundary center. , This represents the soft boundary label at sampling time t, and its value is set to an exponential decay form: , It is the boundary center. , It's half the width of a window. The attenuation rate;

[0046] Step B2: During training, multi-task joint optimization is used, and binary cross-entropy (BCE) is used to calculate the classification loss. and boundary loss : , , , Let represent the writing probability and boundary probability at sampling time t, respectively. This represents the point-by-point writing probability output by the classification branch. The set constituted Indicates that it is a point-by-point hard label The set constituted This represents the point-by-point boundary probability output by the auxiliary boundary branch. The set constituted Indicates the use of point-by-point soft boundary labels The set constituted This indicates that binary cross-entropy is used for calculation, and the time difference of the classification branch output is also included. Normalized to boundary distribution Output the auxiliary boundary branch ( Normalized to boundary distribution Then, the difference between the two is measured using a one-dimensional Wasserstein distance to obtain the alignment loss. : , This involves calculating the one-dimensional Wasserstein distance and finally, calculating the total loss. , These are weighting coefficients used to balance the influence of the alignment terms.

[0047] Classification loss Point-by-point writing probability used for supervising the output of classification branches With point-by-point hard labels Consistency, boundary loss Pointwise boundary probabilities used to supervise the output of auxiliary boundary branches With soft boundary labels Consistency, alignment loss This is used to align the "mutation location of the classification sequence" with the "high response location of the boundary branch" on the time axis, thereby reducing boundary jitter and mis-strokes in online inference.

[0048] (2) Cross-domain meta-learning: This method employs the concept of Model-Agnostic Meta-Learning (MAML), treating the "user" as a task to train a rapidly adaptable base model that transitions from pinch-and-release to continuous input. The specific implementation process is as follows:

[0049] For each user task, a support set and a query set are constructed. The support set is taken from the user's self-labeled pinch-release data (large in scale and easy to collect), while the query set is taken from the user's continuous handwritten input data (non-pinch-release gestures), which has been manually labeled (small in scale and costly), thereby improving performance under real continuous input data during inference. The meta-model parameters are set as follows. First, the meta-loss of each user task in each training batch is calculated. Then, the gradients of the meta-losses of all user tasks within the same training batch are accumulated, and meta-updates are performed accordingly. This achieves a model initialization that can generalize across users and quickly personalize for new users. The user task... The calculation process of the meta-loss is as follows: First, in the user task... Support set The above is used to adapt and obtain temporary parameters for the task. Subsequently in user tasks query set The meta-loss of the above task is calculated, and the meta-update process is set as follows: , The meta-learning rate, This represents the cumulative sum of the meta-losses for all user tasks within the current training batch. The total cumulative meta-loss represents the difference between the meta-parameters. gradient, This indicates that the meta-parameters are updated by gradient descent along the gradient direction of the total meta-loss. The value is the total loss obtained from boundary-aware supervision. .

[0050] When deploying the system, new users can directly use the parameters upon logging in. As a cold start model, and by collecting the user's pinch-release gesture self-labeled data for rapid fine-tuning, a personalized user model is obtained.

[0051] In step three, the present invention can further include using time-series windows as the input unit of the deep separable temporal convolutional network in the lightweight neural network model during inference applications, instead of classifying point by point independently, and further includes:

[0052] Step C1: Set the target output segment length to The length of the historical context is The future context length is The input window of a depthwise separable temporal convolutional network is expanded to: , This represents the expanded sequence consisting of all kinematic eigenvectors within the time series window. … , … , … Representing time respectively … , … , … The kinematic eigenvectors;

[0053] Step C2: Depthwise separable temporal convolutional networks with strides scroll, This is set according to actual business needs, pending future contextual requirements. Then output the target interval The writing probability sequence is composed of the writing probabilities at all times within the target interval.

[0054] With step size Scrolling will temporarily store the output until the context is satisfied in the future. Submitting the final result for the corresponding interval later can generate a fixed-delay output. The sliding window provides more sufficient historical and future evidence for the boundary, thereby improving the accuracy of target interval identification.

[0055] Fixed-delay inference introduces perceptible latency. Figure 2 This diagram illustrates the process of inference via a sliding window and the display using an asynchronous visualization strategy, as described in this invention. To avoid a lack of immediate feedback for the user while writing, this invention employs an asynchronous visualization strategy, further including:

[0056] Inference is performed in a separate thread to avoid blocking UI refresh; the UI thread renders semi-transparent trajectory guide lines in real time to reflect the original hand movements; after the model outputs confirmed strokes, the guide lines are covered and hidden with stable strokes, and strokes are submitted segment by segment through the output queue according to the UI refresh frequency to avoid sudden changes in results.

[0057] like Figure 3 As shown, the present invention relates to a continuous air handwriting recognition system for extended reality, comprising:

[0058] The data acquisition device samples over time to obtain the three-dimensional position sequence of the user's finger key points during writing, as well as the 6-DOF pose information of the XR computing device at the corresponding sampling time. The three-dimensional position sequence of the finger key points is represented as follows: , Indicates time Key points of the fingers, This refers to the number of sampling times; the XR computing device operates at the sampling time. The 6-DOF pose information is derived from the rotation matrix. With translation vector Characterization;

[0059] The neural network model device constructs and trains a lightweight neural network model. The workflow of the lightweight neural network model is as follows: the input data is preprocessed and corresponding kinematic features are constructed. Then, the constructed kinematic features are passed through a deep separable temporal convolutional network to output the writing probability and boundary probability at each sampling time. The boundary probability is used for auxiliary supervision during training.

[0060] The inference application device uses a trained lightweight neural network model to infer the writing probability at the sampling time from the sampled data, and extracts connected segments according to a threshold to recover the stroke segment set for subsequent real-time rendering, stroke-level editing and subsequent standard character recognition, thereby realizing visual interaction with the user.

[0061] To verify the effectiveness and usability of this technical solution in real XR scenarios, the following experimental verifications and implementation parameter settings were also conducted:

[0062] A prototype system was implemented based on the Meta Quest 3 mixed reality headset. The interface and interactions were implemented in the Unity engine, and hand skeleton tracking used the hand tracking toolkit from the Meta Quest Unity SDK. Training and data processing were performed using PyTorch. The headset... The model is sampled and inference is run at a refresh rate of Hz. The model inference is exported to ONNX format and integrated into Unity via the OnnxRuntime inference library. Except for training, all runtime phases are executed independently on the headset.

[0063] The key implementation parameter is set as follows: soft tab window size 5 (i.e. =2), Attenuation rate Sliding window parameters , , The DS-TCN backbone uses eight layers of depth-separable temporal convolutional modules stacked together, and is configured with receptive fields of different time scales to cover the speed changes in writing motions.

[0064] In terms of latency, under this setting, the average data buffering time is approximately 439.6ms (standard deviation of approximately 10.2ms), the average model inference time is approximately 2.39ms (standard deviation of approximately 1.0ms), and the overall fixed latency is approximately 0.45s.

[0065] In offline evaluation, frame-by-frame discrimination and stroke segment recovery were quantitatively verified. At the frame-by-frame level, thresholds were used to... Turn to The system reports the original frame-level accuracy and the fault-tolerant accuracy within the boundary tolerance range to reflect boundary ambiguity. At the stroke segment level, it uses Intersection-over-Union (IoU) based matching and reports the segment-level F1 score: F1@{0.25,0.5,0.75}.

[0066] This application compared the XR aerial writing paradigm with two common paradigms in subject experiments: virtual-plane-based distance triggering and pinch-release-based explicit on / off control. The comparative experiments included offline quantitative assessment and online closed-loop assessment (one set). Comparative user studies, and a set of Online closed-loop writing experiment). Figure 4 The comparison of offline quantitative results for different stroke recognition methods is shown. Figure 5 The results of the comparative experiment are shown. Figure 6 The NASA-TLX scale results from an online user study are presented. The results show that this application, while maintaining accuracy in continuous air handwriting, outperforms solutions requiring frequent explicit switching or continuous depth control in terms of writing efficiency and tactile burden. Furthermore, asynchronous visualization effectively alleviates the waiting feeling caused by fixed latency. Since various XR models follow common kinematic principles in spatial positioning and gesture keypoint setting, the technical effects of this solution can be similarly extended to other XR platforms.

[0067] See Figure 7 , Figure 7 This is a structural block diagram of a computing device 700 illustrated in an exemplary embodiment of this specification. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0068] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0069] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0070] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 700 can also be a mobile or stationary server or cloud server, etc.

[0071] The processor 720 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described method for continuous air handwriting recognition for extended reality.

[0072] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described continuous aerial handwritten stroke recognition method for extended reality belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described continuous aerial handwritten stroke recognition method for extended reality.

[0073] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described method for continuous air handwriting recognition in extended reality.

[0074] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the above-described method for continuous aerial handwriting recognition in extended reality. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-described method or system for continuous aerial handwriting recognition in extended reality.

[0075] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described method for continuous aerial handwriting recognition in extended reality.

[0076] The above is an illustrative scheme of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the above-described method for continuous aerial handwriting recognition in extended reality belong to the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the above-described method or system for continuous aerial handwriting recognition in extended reality.

[0077] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0078] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0079] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for continuous aerial handwritten stroke recognition in extended reality, characterized in that, Including: Step 1: Sampling by time to obtain the three-dimensional position sequence of key finger points when the user writes, and the 6-DOF pose information of the XR computing device at the corresponding sampling time; Step 2: Construct and train a lightweight neural network model. The workflow of the lightweight neural network model is as follows: preprocess the input data and construct the corresponding kinematic features. Then, pass the constructed kinematic features through a deep separable temporal convolutional network to output the writing probability and boundary probability at each sampling time. The boundary probability is used for auxiliary supervision during training. Step 3: The sampled data is processed through a trained lightweight neural network model to infer and output the writing probability at the sampling time. Connected segments are extracted based on the threshold to recover the stroke segment set, thereby realizing visual interaction with the user.

2. The method according to claim 1, characterized in that, In step one, the key point position of the fingers is set as the average point of the fingertips of the thumb and index finger.

3. The method according to claim 1, characterized in that, It also includes labeled training data, which includes: The pinch-release gesture interaction state is used as a supervision signal to collect self-labeled training data. That is, when the gesture is in the pinch state, the collected key point positions are labeled as writing state, and when the gesture is in the release state, the collected key point positions are labeled as non-writing movement.

4. The method according to claim 1, characterized in that, The workflow of a lightweight neural network model is as follows: Step A1, Preprocessing: Based on the camera pose of the center frame of the window, transform the key point position of each moment in the three-dimensional position sequence of the finger key points from the world coordinate system to the camera coordinate system of the XR computing device, and normalize the scale within the window according to the maximum span of the trajectory points within the window. Step A2: Constructing kinematic features: Convert the keypoint positions at each moment in the 3D position sequence of finger keypoints into kinematic feature vectors. , The kinematic feature vector represents time t. The kinematic features include the position of the trajectory point and its change over time, including velocity, tangential acceleration, and normal acceleration. Step A3: Input the kinematic feature vector into a depthwise separable temporal convolutional network. This network consists of several stacked depthwise separable temporal convolutional modules. Each module includes: one-dimensional depthwise convolution, non-linear activation, pointwise convolution, and residual connection. The network adopts a dual-branch output: a classification branch and an auxiliary boundary branch. The classification branch outputs the writing probability at each time step, which is used to determine whether each point belongs to the starting stroke. The auxiliary boundary branch outputs the state transition boundary probability at each time step, which is used to explicitly strengthen the localization of the stroke start and end boundaries.

5. The method according to claim 1, characterized in that, In step two, the training strategy for the lightweight neural network model consists of two parts. composition: (1) Boundary-aware supervision, the specific implementation process is as follows: Step B1: Obtain point-by-point hard labels: , This represents the label at sampling time t. Indicates the writing state. To represent non-writing movement, the state transition time of the label is set as the boundary center, and all boundary centers constitute a boundary center set. Then, a soft boundary label is constructed within a preset area of ​​each boundary center: , This represents the soft boundary label at sampling time t, and its value is set to an exponential decay form: , It is the boundary center. , It's half the width of a window. The attenuation rate; Step B2: During training, multi-task joint optimization is used, and binary cross-entropy is used to calculate the classification loss. and boundary loss : , , , Let represent the writing probability and boundary probability at sampling time t, respectively. This represents the point-by-point writing probability output by the classification branch. The set constituted Indicates that it is a point-by-point hard label The set constituted This represents the point-by-point boundary probability output by the auxiliary boundary branch. The set constituted Indicates a point-by-point soft boundary label The set constituted This indicates that binary cross-entropy is used for calculation, and the temporal difference of the classification branch output is normalized to the boundary distribution. The auxiliary boundary branch output is normalized to the boundary distribution. Then, the difference between the two is measured using a one-dimensional Wasserstein distance metric to obtain the alignment loss. : , This involves calculating the one-dimensional Wasserstein distance and finally, calculating the total loss. , These are weighting coefficients used to balance the influence of the alignment term. (2) Cross-domain meta-learning, the specific implementation process is as follows: For each user task, a support set and a query set are constructed. The support set is taken from the user's self-annotated pinch-release gesture data, and the query set is taken from the user's continuously input handwritten data, which has been manually labeled. The meta-model parameters are set as follows. First, the meta-loss of each user task in each training batch is calculated. Then, the gradients of the meta-losses of all user tasks within the same training batch are accumulated, and meta-updates are performed accordingly, thus completing the model initialization. (The user task...) The calculation process of the meta-loss is as follows: First, in the user task... Support set The above is used to adapt and obtain temporary parameters for the task. Subsequently in user tasks query set The meta-loss of the above task is calculated, and the meta-update process is set as follows: , The meta-learning rate, This represents the cumulative sum of the meta-losses for all user tasks within the current training batch. The total cumulative meta-loss represents the difference between the meta-parameters. gradient, This indicates that the meta-parameters are updated by gradient descent along the gradient direction of the total meta-loss. The value is the total loss obtained from boundary-aware supervision. .

6. The method according to claim 1, characterized in that, In step three, during inference applications, the time-series window is used as the input unit for the deep separable temporal convolutional network in the lightweight neural network model, further including: Step C1: Set the target output segment length to The length of the historical context is The future context length is The input window of a depthwise separable temporal convolutional network is expanded to: , This represents the expanded sequence consisting of all kinematic eigenvectors within the time series window. … , … , … Representing time respectively … , … , … The kinematic eigenvectors; Step C2: Depthwise separable temporal convolutional networks with strides Scroll, pending future context. Then output the target interval The writing probability sequence is composed of the writing probabilities at all times within the target interval.

7. The method according to claim 1, characterized in that, An asynchronous visualization strategy is adopted, including: Inference is performed in a separate thread to avoid blocking UI refresh; the UI thread renders semi-transparent trajectory guide lines in real time to reflect the original hand movements; after the model outputs confirmed strokes, the guide lines are covered and hidden with stable strokes, and strokes are submitted segment by segment through the output queue according to the UI refresh frequency to avoid sudden changes in results.

8. A continuous air-based handwriting stroke recognition system for extended reality, characterized in that, Including: The data acquisition device samples over time to obtain the three-dimensional position sequence of key finger points when the user writes, as well as the 6-DOF pose information of the XR computing device at the corresponding sampling time; The neural network model device constructs and trains a lightweight neural network model. The workflow of the lightweight neural network model is as follows: the input data is preprocessed and corresponding kinematic features are constructed. Then, the constructed kinematic features are passed through a deep separable temporal convolutional network to output the writing probability and boundary probability at each sampling time. The boundary probability is used for auxiliary supervision during training. The inference application device uses a trained lightweight neural network model to infer the writing probability at the sampling time from the sampled data, and extracts connected segments based on a threshold to recover the stroke segment set, thereby realizing visual interaction with the user.

9. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the continuous air handwriting stroke recognition method for extended reality as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the continuous air handwriting recognition method for extended reality as described in any one of claims 1-7.

Images