Air handwriting recognition method based on multi-modal data integration under monocular camera

Abstract

The application relates to an aerial handwriting recognition method based on multi-modal data integration under a monocular camera, multi-modal data including image data and time series data; the multi-modal data is fused by using a multi-modal data fusion model to obtain fused 3D data; the 3D data is processed by using a data projection method to obtain three characteristic projection results, namely projection data 1, projection data 2 and projection data 3, and the three characteristic projections are all 2D images; the three obtained projection characteristics are respectively learned to obtain three base learners; the results obtained by the three base learners are integrated by using an integration method based on a main learner to obtain a final recognition result. The aerial handwriting recognition method based on multi-modal data integration under the monocular camera can greatly improve the recognition rate by integration learning, and the recognition efficiency can be improved by 10% in experiments. The method transforms multi-modal data of time data and image data into multiple orthogonal homogeneous data sets, and then participates in the integration learning method, and the method is simple but achieves good recognition effect. The method can be applied to aerial handwriting or similar time series learning tasks, and is practical.

Description

Technical Field

[0001] This invention relates to the field of handwriting recognition technology, specifically to an aerial handwriting recognition method based on multimodal data integration using a monocular camera. Background Technology

[0002] Air handwriting (or simply air writing) refers to users writing text in a virtual input area using gestures, enabling computers to recognize characters like humans. It is a truly natural form of human-computer interaction. Air handwriting is an important input method for intelligent human-computer interaction, improving communication and interaction between computers and users. It provides a convenient operating method in specialized fields such as service robots, robot vision navigation, mobile terminals, smart home appliances, and smart cars.

[0003] Existing methods for air handwriting recognition include those based on data gloves, rings, surface electromyography (EMG) signals, and mobile terminal accelerometers, but all require wearable devices, significantly reducing the user experience. Methods based on depth cameras like Kinect and Leap, binocular vision, and multiple sensors can accurately capture hand movements and shapes, but these devices are expensive, increasing system implementation costs and making them impractical. Air handwriting recognition based on monocular vision can be used with ordinary mobile devices equipped with cameras to achieve barrier-free, free interaction in any scenario, representing the current mainstream of gesture interaction and the data acquisition method used in this patent.

[0004] Traditional air-written isolated word recognition processes consist of three parts: preprocessing, feature extraction, and classifier design. These three stages are interconnected and mutually supportive. Preprocessing aims to eliminate noise and normalize characters, and its impact on recognition accuracy is significant. Feature extraction aims to extract as many features as possible that describe the intra-class and inter-class differences of characters, such as chain code features, moment features, gradient features, local contour features, linear eccentricity features, flatness, and ellipse area features. Classifier design involves selecting appropriate classifiers for different features, including dynamic time warping, support vector machines, and hidden Markov models. Traditional methods are easy to understand and can achieve some simple air-written handwriting recognition, but they lack versatility and require extensive experience and multiple trials to select good features. When the gesture characters are expanded and the model becomes more complex, recognition accuracy often decreases.

[0005] With the advent of the big data era and the rise of deep learning technology, classifiers have been largely replaced by various cleverly designed deep neural networks. Furthermore, relying on the powerful feature extraction capabilities of neural networks layer by layer, feature extraction can be performed; end-to-end recognition systems can also be used for direct recognition, eliminating the need for complex preprocessing and feature extraction. Convolutional neural networks (CNNs) have achieved good results in handling many classification problems. For example, the classic LeNet-5, designed by LeCun et al., is a handwritten character recognition system based on a convolutional neural network, used to recognize the MNIST handwritten image dataset, achieving a recognition efficiency of up to 99% after improvements. Later, researchers analyzed the feature extraction performance of CNNs based on the characteristics of different handwritten characters and improved CNNs by incorporating time-series information, achieving better results with methods such as Siamese networks, RNNs, CNN+LSTM, and RNN+LSTM. Therefore, using deep learning methods for aerial handwriting recognition has become an inevitable trend.

[0006] While CNNs demonstrate excellent detection performance on existing datasets, such as MNIST, achieving a recognition rate as high as 99%, overfitting can occur when training data is insufficient, leading to decreased model performance. Existing datasets for aerial handwriting recognition are relatively small, and creating large-scale datasets is difficult, time-consuming, and labor-intensive. The project team's self-made aerial handwritten digit isolated word dataset has the following dimensions: 3 people * 10 digits * 10 samples. With 3 * 10 samples in the training set, 3 * 10 samples in the validation set, and 24 * 10 samples in the test set, LeNet-5 can only achieve a recognition rate of 84%, and it can only utilize image features. Some literature extracts traditional trajectory features for recognition, but this also fails to fully utilize multimodal data, resulting in low recognition rates on small samples.

[0007] Therefore, it is necessary to design an aerial handwriting recognition method based on multimodal data integration using a monocular camera to address the above problems. Summary of the Invention

[0008] The purpose of this invention is to provide an aerial handwriting recognition method based on multimodal data integration using a monocular camera, in order to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: an aerial handwriting recognition method based on multimodal data integration under a monocular camera, wherein the multimodal data includes image data and temporal data; the multimodal data is fused using a multimodal data fusion model to obtain fused 3D data; the 3D data is processed using a data projection method to obtain three feature projection results, namely projection data 1, projection data 2, and projection data 3, all of which are 2D images; the three projection features are learned separately to obtain three base learners; the results obtained from the three base learners are integrated using an ensemble method based on the master learner to obtain the final recognition result.

[0010] Furthermore, the specific steps of the aerial handwriting recognition method based on multimodal data integration under a monocular camera are as follows: a. acquisition of multimodal data; b. multimodal data fusion model; c. data projection transformation; d. learner; e. integration method based on master learner; f. recognition result.

[0011] Furthermore, the specific steps for acquiring multimodal data are as follows: Aerial handwriting recognition is a trajectory-based gesture sequence recognition method. It ignores changes in gesture shape and uses the gesture's motion trajectory as the analysis data. The data acquisition process can be divided into two parts: first, preprocessing the gesture frames in the video stream, including gesture segmentation and gesture tracking. Gesture tracking can be based on prior information obtained from gesture segmentation; then, based on the tracking results, data extraction is performed on the gesture's motion trajectory. The extracted data includes the gesture's position on the writing plane (2D data) and the time sequence number of each frame (temporal features, 1D data). The resulting image data and temporal data constitute the acquired multimodal data.

[0012] Furthermore, regarding the b. multimodal data fusion model: the obtained image data differs from the time-series data type, constituting multimodal data. The multimodal data fusion model is used to fuse it into 3D data, as follows: the original image data is 2D data, containing x and y coordinates, which are used as the x and y coordinates of the fused 3D data. The time-series data is used as the z-coordinate of the fused 3D data, thus obtaining the fused 3D data.

[0013] Furthermore, in step c, data projection transformation: the fused 3D data is projected onto three orthogonal coordinate planes using a feature projection method, resulting in three distinct datasets. The method is as follows: assuming the three coordinate axes of the fused 3D data are x, y, and z, projections of the 3D data onto the xy, xz, and yz planes are taken respectively, resulting in 2D data. By reducing the dimensionality of the 3D data through projection, three mutually orthogonal 2D datasets are obtained. This data transformation fully extracts data features, thereby improving the final recognition efficiency.

[0014] Furthermore, regarding the d. learner: three base learners are obtained by training the three 2D data points respectively. The method is as follows: first, the 2D data is scaled to a size of 28*28, and then the same base learner is used for learning. The base learner used is an 8-layer CNN. The 8 layers of the network, from front to back, are: an input layer of 28*28, 16 5*5 convolutional layers, a max pooling layer, 36 5*5 convolutional layers, a max pooling layer, a flattening layer, a hidden layer with 128 neurons, and an output layer.

[0015] Furthermore, regarding e. the ensemble method based on the master learner: the three base learners are integrated to achieve significantly superior generalization performance compared to a single learner. This integration uses a voting method based on the master learner. Specifically, since the data projected onto the xy-plane represents handwritten semantic content in the air, the learner learned from the xy-plane projection data is used as the master learner, and the learners learned from the xz-plane and yz-plane projection data are used as auxiliary learner 1 and auxiliary learner 2, respectively. The voting method is as follows: if the recognition results of auxiliary learner 1 and auxiliary learner 2 are the same, the final recognition result is the recognition result of the auxiliary learner; if the recognition results of auxiliary learner 1 and auxiliary learner 2 are different, the final recognition result is the recognition result of the master learner.

[0016] Furthermore, the f. recognition result: The final recognition result is the semantics of handwritten characters in the air, such as the numbers "0-9".

[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: This method is an aerial handwriting recognition method based on multimodal data ensemble under a monocular camera. It innovatively transforms multimodal data to obtain multiple orthogonal homogeneous datasets, which are then used in ensemble learning. One-dimensional time-series features and two-dimensional writing trajectory image features are fused into three-dimensional features. These three-dimensional features are then projected onto three projection planes, each serving as different data for the learning model to recognize. Finally, the three results are voted to obtain the final result. The innovation of this method lies in its ability to significantly improve the recognition rate through ensemble learning, achieving a 10% improvement in recognition efficiency in experiments. The method's innovation lies in transforming multimodal data (time data and image data) to obtain multiple orthogonal homogeneous datasets before using ensemble learning. The method is simple yet achieves good recognition results. This method can be applied to aerial handwriting or similar time-series learning tasks, demonstrating strong practicality. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the ensemble learning method for multimodal data transformation in the aerial handwriting recognition method based on multimodal data ensemble under a monocular camera according to the present invention.

[0019] Figure 2 This is a 3D data image of the fused handwritten digit "9" in the air using the aerial handwriting recognition method based on multimodal data integration under a monocular camera according to the present invention.

[0020] Figure 3 This is a data diagram of the xy-plane projection of the aerial handwriting recognition method based on multimodal data integration under a monocular camera according to the present invention;

[0021] Figure 4 This is a data diagram of the yz-plane projection of the aerial handwriting recognition method based on multimodal data integration under a monocular camera according to the present invention;

[0022] Figure 5 This is a data diagram of the xz-plane projection of the aerial handwriting recognition method based on multimodal data integration under a monocular camera according to the present invention;

[0023] Figure 6 This diagram illustrates the master learner-based ensemble method for the aerial handwriting recognition method based on multimodal data ensemble under a monocular camera, as described in this invention.

[0024] Figure 7 This is a confusion matrix diagram of the CNN-based aerial handwriting recognition method using a monocular camera and multimodal data integration for image feature extraction and recognition.

[0025] Figure 8 This is a confusion matrix diagram of the aerial handwriting recognition method based on multimodal data integration under a monocular camera according to the present invention.

[0026] Figure 9 These are some aerial handwriting video frames from the aerial handwriting recognition method based on multimodal data integration using a monocular camera, as described in this invention.

[0027] Figure 10 This is a schematic diagram of the isolated handwritten words "0-9" in the air using the air handwriting recognition method based on multimodal data integration under a monocular camera according to the present invention. Detailed Implementation

[0028] like Figures 1 to 10 As shown, this invention provides a technical solution: an aerial handwriting recognition method based on multimodal data integration under a monocular camera. This method inputs aerial handwriting multimodal data, which includes image data and temporal data. The multimodal data is fused using a multimodal data fusion model to obtain fused 3D data. The 3D data is processed using a data projection method to obtain three feature projection results: projection data 1, projection data 2, and projection data 3, all of which are 2D images. The three projection features are learned separately to obtain three base learners. An ensemble method based on the master learner is used to integrate the results obtained from the three base learners to obtain the final recognition result.

[0029] The specific steps are as follows:

[0030] a. Acquisition of multimodal data

[0031] Aerial handwriting recognition is a trajectory-based gesture sequence recognition method that ignores changes in gesture shape and uses the gesture's motion trajectory as the analysis data. The data acquisition process can be divided into two parts: first, gesture preprocessing is performed on the gesture frames in the video stream, including gesture segmentation and gesture tracking. Gesture tracking can be based on prior information obtained from gesture segmentation; then, based on the tracking results, the motion trajectory of the gesture is extracted. The extracted data includes the gesture's position on the writing plane (2D data) and the time sequence number of each frame (temporal features, 1D data). The resulting image data and temporal data constitute the acquired multimodal data.

[0032] b. Multimodal data fusion model

[0033] The obtained image data differs from the time-series data type, representing multimodal data. A multimodal data fusion model is used to fuse it into 3D data, as follows: the original image data is 2D data with x and y coordinates, which are used as the x and y coordinates of the fused 3D data. The time-series data is used as the z-coordinate of the fused 3D data, thus obtaining the fused 3D data.

[0034] Multimodal data fusion was performed on the handwritten digit "9" in the air to obtain the fused 3D data. Figure 2 .

[0035] c. Data projection transformation

[0036] The fused 3D data is projected onto three orthogonal coordinate planes using a feature projection method, resulting in three distinct datasets. The method is as follows: Assuming the three coordinate axes of the fused 3D data are x, y, and z, projections of the 3D data onto the xy, xz, and yz planes are taken respectively, resulting in 2D data. By projecting the 3D data to reduce its dimensionality, three mutually orthogonal 2D datasets are obtained. This data transformation method fully extracts data features, thereby improving the final recognition efficiency.

[0037] A data projection transformation was performed on the handwritten digit "9" in the air, resulting in three orthogonal 2D data sets. Figure 3-5 .

[0038] d. Learning device

[0039] Three base learners were obtained by training on the three 2D data points. The method is as follows: First, the 2D data was scaled to a size of 28*28, and then the same base learner was used for learning. The base learner used was an 8-layer CNN. The 8 layers of the network, from front to back, are: 28*28 input layer, 16 5*5 convolutional layers, max pooling layer, 36 5*5 convolutional layers, max pooling layer, flattening layer, 128-neuron hidden layer, and output layer.

[0040] e. Ensemble method based on master learner

[0041] The three base learners are integrated to achieve significantly better generalization performance than a single learner. This integration uses a voting method based on the master learner; the process is detailed below. Figure 6 The specific method is as follows: because the data projected onto the xy-plane represents handwritten semantic content in the air, the learner learned from the xy-plane projection data is used as the primary learner, and the learners learned from the xz-plane and yz-plane projection data are used as secondary learner 1 and secondary learner 2, respectively. The voting method is as follows: if the recognition results of secondary learner 1 and secondary learner 2 are the same, then the final recognition result is the recognition result of the secondary learner; if the recognition results of secondary learner 1 and secondary learner 2 are different, then the final recognition result is the recognition result of the primary learner.

[0042] f. Recognition Results

[0043] The final recognition result is the semantics of handwritten characters in the air, such as the numbers "0-9".

[0044] This method is an aerial handwriting recognition approach based on multimodal data ensemble using a monocular camera. It innovatively transforms multimodal data to obtain multiple orthogonal, homogeneous datasets, which are then used in ensemble learning. One-dimensional time-series features are fused with two-dimensional writing trajectory image features to form three-dimensional features. These three-dimensional features are then projected onto three different projection planes, each serving as distinct data for the learning model. Finally, the three results are voted to obtain the final result. The innovative aspect of this method lies in its ability to significantly improve the recognition rate through ensemble learning, achieving a 10% improvement in recognition efficiency in experiments. The method's innovation lies in its transformation of multimodal time and image data into multiple orthogonal, homogeneous datasets before ensemble learning. While simple, the method achieves good recognition results. This approach can be applied to aerial handwriting recognition or similar time-series learning tasks, demonstrating strong practicality.

[0045] Implementation example: This method is applied to the recognition of handwritten isolated digits "0-9" in the air.

[0046] In practice, this method was applied to the recognition of isolated words "0-9" in air handwriting recognition. Handwriting videos were captured using a regular mobile phone camera. After preprocessing, a dataset of 3 people * 10 digits * 10 samples was obtained. This method was then used to recognize the multidimensional data. 3 samples were used as the training set, 3 samples as the validation set, and 24 samples as the test set. A recognition rate of 95% was achieved on the test set.

[0047] During implementation, a handwritten dataset of isolated words "0-9" in the air was established. Some video frames of the air handwritten characters in the dataset are available in the dataset. Figure 9 .

[0048] The method used for data preprocessing to obtain the air writing data is as follows: First, before gesture segmentation, keyframes are extracted from the gesture image sequence to obtain a very small number of image sequences for subsequent recognition. This greatly reduces the running time of the gesture recognition system. Then, image preprocessing is performed. During image preprocessing, each frame in the video stream is subjected to image processing techniques such as smoothing, denoising, color balancing, color space conversion, morphological filtering, and image binarization to segment the gesture region. Then, the centroid of the segmented gesture region is calculated as the current writing position. The writing position (x, y) coordinates are 2D image data, and the keyframe sequence number is a 1D temporal feature.

[0049] See the 2D image data for obtaining the digits "0-9" Figure 10 .

[0050] Using this method, a recognition rate of 95% can be achieved. The confusion matrix is ​​shown below. Figure 6 This method achieves improved recognition rates compared to CNN algorithms and traditional feature extraction methods on small sample sizes.

[0051] Working Principle: Aerial handwriting recognition is a trajectory-based gesture sequence recognition method. It ignores changes in gesture shape and uses the gesture's trajectory as the analysis data. The data acquisition process can be divided into two parts: First, gesture preprocessing is performed on the gesture frames in the video stream, including gesture segmentation and gesture tracking. Gesture tracking can be based on prior information obtained from gesture segmentation. Then, based on the tracking results, the motion trajectory of the gesture is extracted. The extracted data includes the gesture's position on the writing plane (2D data) and the time sequence number of each frame (temporal feature, 1D data). The obtained image data and temporal data types constitute the acquired multimodal data. A multimodal data fusion model is used to fuse this data into 3D data. The method is as follows: The original image data is 2D data with x and y coordinates, which are used as the x and y coordinates of the fused 3D data. The temporal data is used as the z-coordinate of the fused 3D data. This yields the fused 3D data. Multimodal data fusion is performed on the handwritten digit "9" in the air to obtain the fused 3D data. Figure 2 The fused 3D data is projected onto three orthogonal coordinate planes using a feature projection method, resulting in three distinct datasets. The method is as follows: Assuming the three coordinate axes of the fused 3D data are x, y, and z, projections of the 3D data onto the xy, xz, and yz planes are taken respectively, resulting in 2D data. Dimensionality reduction is achieved through 3D data projection, yielding three mutually orthogonal 2D datasets. This data transformation fully extracts data features to improve the final recognition efficiency. The data projection transformation is applied to the handwritten digit "9" in the air, resulting in three transformed orthogonal 2D datasets (see [example example]). Figure 3-5 Three base learners were trained from the three obtained 2D data sets. The method is as follows: First, the 2D data was scaled to a size of 28*28, and then the same base learner was used for training. The base learner used was an 8-layer CNN, with the 8 layers from front to back being: an input layer of 28*28, 16 5*5 convolutional layers, a max pooling layer, 36 5*5 convolutional layers, a max pooling layer, a flattening layer, a hidden layer with 128 neurons, and an output layer. The three base learners were then integrated to achieve significantly better generalization performance than a single learner. The integration method used was a voting method based on the master learner. The process is described in [link to process]. Figure 6The specific method is as follows: because the data projected onto the xy-plane represents handwritten semantic content in the air, the learner learned from the xy-plane projection data is used as the primary learner, and the learners learned from the xz-plane and yz-plane projection data are used as secondary learner 1 and secondary learner 2, respectively. The voting method is as follows: if the recognition results of secondary learner 1 and secondary learner 2 are the same, then the final recognition result is the result of the secondary learner; if the recognition results of secondary learner 1 and secondary learner 2 are different, then the final recognition result is the result of the primary learner. The final recognition result represents handwritten semantic content in the air, such as the numbers "0-9".

Claims

1. A method for aerial handwriting recognition based on multimodal data fusion using a monocular camera, characterized in that, The method includes the following steps in sequence: a) Acquisition of multimodal data: Acquiring two different types of multimodal data synchronously extracted after gesture preprocessing, including two-dimensional image data corresponding to the aerial handwriting process captured by a monocular camera and time-series data of the frames of the aerial handwriting process, i.e., time sequence numbers; b) Multimodal data fusion model processing: Using a multimodal data fusion model to fuse the two-dimensional image data and time-series data to obtain fused three-dimensional data. The fusion method is to use the x and y coordinates of the original two-dimensional image data as the x and y coordinates of the fused three-dimensional data, and use the time-series data as the z coordinate of the fused three-dimensional data; c) Data projection transformation: Processing the three-dimensional data using a feature projection method to obtain the projection results of three two-dimensional features; d) Base learner training: Learning the three projection features obtained respectively to obtain three base learners; e) Integration processing based on the master learner and obtaining recognition results: Integrating the results obtained from the three base learners using an integration method based on the master learner to obtain the final recognition result of the aerial handwriting.

2. The aerial handwriting recognition method based on multimodal data integration using a monocular camera according to claim 1, characterized in that, The specific steps for acquiring the multimodal data are as follows: Air handwriting recognition is a trajectory-based gesture sequence recognition that ignores changes in gesture shape and uses the motion trajectory of the gesture as the analysis data. The data acquisition process includes gesture preprocessing, gesture segmentation, and gesture tracking of gesture frames in the video stream. Gesture tracking is achieved based on prior information obtained from gesture segmentation. Then, based on the tracking results, the motion trajectory of the gesture is extracted. The extracted data includes the position of the gesture on the writing plane, which is two-dimensional data, and the time sequence number of each frame, which is a temporal feature and is one-dimensional data. The resulting image data and temporal data constitute the multimodal data.

3. The aerial handwriting recognition method based on multimodal data integration under a monocular camera according to claim 1, characterized in that, The specific method for data projection transformation (c) is as follows: the fused 3D data is projected onto three orthogonal coordinate planes using a feature projection method to obtain three different datasets. Further, assuming the three coordinate axes of the fused 3D data are x, y, and z, projections are taken onto the xy plane, xz plane, and yz plane respectively to complete the dimensionality reduction of the 3D data projection, resulting in three mutually orthogonal 2D projection data. By performing data transformation in this way, data features are fully extracted, improving the final recognition efficiency.

4. The aerial handwriting recognition method based on multimodal data integration using a monocular camera according to claim 1, characterized in that, The specific method for training the base learners is as follows: The three obtained two-dimensional projection data are trained and learned separately to obtain three base learners. Specifically, the two-dimensional projection data is first scaled to a size of 28*28, and then trained using the same base learners. The base learners used are 8-layer CNNs. The 8-layer network consists of, from front to back, a 28*28 input layer, 16 5*5 convolutional layers, a max pooling layer, 36 5*5 convolutional layers, a max pooling layer, a flattening layer, a hidden layer with 128 neurons, and an output layer.

5. The aerial handwriting recognition method based on multimodal data integration using a monocular camera according to claim 1, characterized in that, The specific method for integrating the three base learners and obtaining the recognition result based on the master learner is as follows: The three base learners are integrated to achieve significantly better generalization performance than a single learner. A voting method based on the master learner is used. Specifically, since the data projected onto the xy plane is handwritten semantic content in the air, the learner trained on this projection data is used as the master learner, and the learners trained on the xz plane and yz plane projection data are used as auxiliary learner 1 and auxiliary learner 2, respectively. The voting method is as follows: if the recognition results of auxiliary learner 1 and auxiliary learner 2 are consistent, then this result is taken as the final recognition result; if the recognition results of auxiliary learner 1 and auxiliary learner 2 are inconsistent, then the recognition result of the master learner is taken as the final recognition result. The final recognition result is handwritten semantic content in the air, such as the numbers "0-9".

Images