A method and device for inputting text written in the air

Abstract

The present invention provides an air writing text input method and device, which uses a handheld writing stick to write text in the air, uses an embedded platform device with a display screen and a camera to collect and display the writing process, and realizes text input including three types of text, including numbers, symbols and English, through target tracking, text recognition and other methods, so as to meet the requirements of contactless and convenient human-computer interaction. It includes: initializing the air writing text input system; holding the writing stick to enter the air area monitored by the camera and moving the writing stick to the tracking frame; using the target tracking model to track the writing stick and determine the position of the current tracking frame; judging whether to switch the input text type; using the writing stick to write text in the air and generate a text image; using the text recognition model to recognize the text image to recognize the text; adding text display; judging whether to clear the added text; clearing the added text; judging whether the input of all text in the text is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition, and in particular to a method and device for inputting text written in the air. Background Art

[0002] Nowadays, smart electronic devices bring convenience to people's daily lives and improve the efficiency of industrial production and the quality of products. However, most of the interaction methods of smart electronic devices use contact input devices, such as keyboards, mice, touch screens, etc. The contact input method of public electronic devices poses a hidden danger to health and safety. Writing text in the air is a new human-computer interaction method that can effectively avoid contact and realize contactless information input.

[0003] Most of the existing air writing text input systems are based on sensors. For example, the air writing trajectory recognition method, device, storage medium and electronic device disclosed in the Chinese patent publication number CN113052078 have some shortcomings, such as: the need to wear sensor equipment, which affects the user experience; and the problem of poor anti-interference ability in complex environments. In order to better solve these problems, the present invention realizes a contactless, convenient, intelligent and stable text input method and device by writing in the air with a handheld writing stick. Summary of the invention

[0004] In order to overcome the shortcomings of the background technology, the present invention provides a method and device for inputting text by writing in the air, which uses a handheld writing stick to write text in the air, utilizes an embedded platform device with a display screen and a camera to collect and display the writing process, and realizes text input including three types of text, namely numbers, symbols and English, through target tracking, text recognition and other methods, so as to meet the needs of contactless and convenient human-computer interaction.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for inputting text written in the air comprises the following steps:

[0007] Step 1: Initialize the air writing text input system;

[0008] Step 2: Hold the writing stick and enter the aerial area monitored by the camera, and move the writing stick into the tracking frame;

[0009] Step 3: Use the target tracking model to track the writing stick and determine the position of the current tracking box;

[0010] Step 4: Determine whether to switch the input text type, if yes, go to step 5, if not, go to step 6;

[0011] Step 5: Switch the input text type;

[0012] Step 6: Use the writing stick to write text in the air and generate a text image;

[0013] Step 7: Use the text recognition model to recognize the text image and identify the text;

[0014] Step 8: Add text display;

[0015] Step 9: Determine whether to clear the added displayed text, if yes, go to step 10, if not, go to step 11;

[0016] Step 10: Clear the added text;

[0017] Step 11: Determine whether the input of all characters in the text is completed, if so, then end, if not, go to step 4.

[0018] Furthermore, in step 1, initializing the air writing text input system includes the following steps:

[0019] Step 1-1: Initialize the operation window and tracking frame on the display screen;

[0020] Step 1-2: Initialize the input text type;

[0021] Step 1-3: Load the text recognition model parameters that match the input text type;

[0022] Step 1-4: Load the target tracking model parameters.

[0023] Furthermore, the layout of the operation window of step 1-1 is described as follows:

[0024] The operation window includes four areas: type, write, execute and display;

[0025] The type area includes three sub-areas: numbers, symbols and English, which are used to switch the input text type;

[0026] The writing zone is the restricted area where writing appears;

[0027] The execution area includes two sub-areas: end and clear. The end sub-area is used to end the text input, and the clear sub-area is used to delete the currently input text.

[0028] The display area is used to display the recognition result of the airwritten text.

[0029] Furthermore, in the steps 1-3, the text recognition model uses a newly designed network, which specifically includes the following:

[0030] The text recognition model includes text images, feature extraction networks, classification networks and output categories. The text image is the input of the model. The text image contains the complete image and binary image to be recognized. The feature extraction network extracts the deep features of the text image. The feature extraction network includes convolution layer, Bi-CNN layer, average pooling layer, Bi-CNN layer and average pooling layer. The convolution layer performs channel expansion on the complete image and binary image of the text image respectively. The Bi-CNN layer consists of parallel conventional convolution and void convolution followed by channel splicing. The first Bi-CNN layer performs conventional convolution on the channel feature map of the binary image of the text image. The first Bi-CNN layer performs dilated convolution on the channel expansion map of the complete text image. The first Bi-CNN layer performs channel splicing on the two feature maps after regular convolution and dilated convolution. The second Bi-CNN layer performs regular convolution and dilated convolution on the feature map of the previous layer respectively. The second Bi-CNN layer performs channel splicing on the two feature maps after regular convolution and dilated convolution. Both average pooling layers perform size compression on the feature maps after channel splicing. The classification network is used for text classification and recognition. The classification network consists of two fully connected layers, and the output category is the result of text image recognition.

[0031] When the input text type is a number, the output category of the text recognition model is 0-9, and the number of categories is 10;

[0032] When the input text type is a symbol, the output category of the text recognition model is the commonly used symbols, and the number of categories is 38;

[0033] When the input text type is English, the output categories of the text recognition model are uppercase and lowercase English characters, and the number of categories is 52;

[0034] The text recognition model adopts a new dual-image parallel input network structure. The text recognition model recognizes three types of written text: numbers, symbols and English. The text recognition model integrates the global features of the complete text image and the local features of the binary text image. The text recognition model reduces the impact of background information on the feature extraction network.

[0035] Furthermore, the target tracking model in steps 1-4 adopts SiamFC network to complete the writing process tracking of the writing stick, support the drawing of the writing trajectory of the writing stick and realize the judgment of the end of writing of the writing stick.

[0036] Furthermore, in step 3, tracking the writing stick using the target tracking model includes the following steps:

[0037] Step 3-1: Read the writing stick template image;

[0038] Step 3-2: extracting the writing stick search image according to the position of the tracking frame;

[0039] Step 3-3: Call the target tracking model SiamFC network to calculate the response map between the template image of the writing stick and the search image;

[0040] Step 3-4: Determine the position of the current tracking frame according to the writing stick response graph.

[0041] Furthermore, in step 4, the process of determining whether to switch the text type is as follows:

[0042] According to the position of the current tracking frame, it is determined whether the writing stick is in the position of the operation window type area. If it is, the input text type needs to be switched. Otherwise, the input text type does not need to be switched.

[0043] Furthermore, in step 5, the process of switching the input text type is as follows:

[0044] According to the position of the current tracking frame, determine that the writing stick is located in one of the three sub-areas of numbers, symbols and English;

[0045] Set the new input text to the type of this subarea;

[0046] Load the text recognition model parameters that match the new input text type.

[0047] Furthermore, in step 6, writing text in the air and generating a text image includes the following steps:

[0048] Step 6-1: Backup the position of the tracking frame;

[0049] Step 6-2: Move the writing stick in the air area corresponding to the writing area of ​​the operation window;

[0050] Step 6-3: Call the target tracking model SiamFC network to determine the trajectory of the current tracking frame relative to the backup tracking frame;

[0051] Step 6-4: Draw the trajectory of the tracking frame directly in the writing area of ​​the operation window;

[0052] Step 6-5: Repeat the above steps to draw the entire moving trajectory of the tracking box during the writing process;

[0053] Step 6-6: Cover the writing stick with your finger;

[0054] Step 6-7: extract the maximum value of the response map in the SiamFC network of the current target tracking model, and end the text writing operation when the maximum value of the response map is less than the threshold;

[0055] Step 6-8: Extract the trajectory drawn in the writing area of ​​the operation window and generate a text image, including a complete image and a binary image.

[0056] Furthermore, the process of adding text display in step 8 is as follows:

[0057] Read the text displayed in the operation window display area;

[0058] Add new recognized words to the text;

[0059] The new text is output to the display area of ​​the operation window for display.

[0060] Furthermore, the process of determining whether to clear the added displayed text in step 9 is as follows:

[0061] Call the target tracking model SiamFC network to determine the position of the current tracking box;

[0062] According to the position of the current tracking frame, it is determined whether the writing stick is in the clearing sub-area of ​​the operation window execution area. If so, the added displayed text needs to be cleared, otherwise, it does not need to be cleared.

[0063] Furthermore, the process of clearing the added displayed text in step 10 is as follows:

[0064] Read the text displayed in the operation window display area;

[0065] Delete the last word in the text;

[0066] The new text is output to the display area of ​​the operation window for display.

[0067] Furthermore, the process of determining whether the input of all characters in the text is completed in step 11 is as follows:

[0068] Call the target tracking model SiamFC network to determine the position of the current tracking box;

[0069] According to the position of the current tracking frame, it is determined whether the writing stick is in the end sub-area of ​​the operation window execution area. If so, the text input is terminated, and if not, the text input can be continued.

[0070] An air writing text input device using the air writing text input method includes the following modules:

[0071] Camera module: used to collect real-time video of writing and operation of the writing stick in the air;

[0072] Tracking module: contains the target tracking model, which is used to track the writing stick, generate text images and end text writing;

[0073] Recognition module: contains a text recognition model, which is used to recognize three types of texts: numbers, symbols and English written in the air;

[0074] Display module: used to display the operation window, video, tracking frame, text image and recognized text;

[0075] Database module: used to store data and parameters of the air writing text input method;

[0076] Embedded module: used for driving the air writing text input device and running the air writing text input method program;

[0077] The embedded module is connected to the camera and the display screen; the embedded module runs the database module to save the program, data and parameters of the written text input method; the embedded module runs the tracking module and the recognition module to complete the tracking and recognition functions of the air-written text input; the embedded module provides GPU computing power for the models of the tracking module and the recognition module.

[0078] Compared with the prior art, the present invention has the following beneficial effects:

[0079] (1) The present invention provides a method and device for inputting text by writing in the air, using a disposable writing stick to write text in the air, completing the action tracking, text generation and end judgment of writing in the air through a target tracking module, and completing the recognition of three types of handwriting text through a text recognition module, thereby realizing the function of inputting text by writing in the air;

[0080] (2) The present invention provides a method and device for inputting text written in the air, which uses a text recognition model to recognize three types of written text: numbers, symbols, and English. The text recognition model adopts a new dual-image parallel input network structure, which integrates the global features of the complete text image and the local features of the binary text image, thereby reducing the impact of background information on the feature extraction network and improving the accuracy and reliability of text recognition.

[0081] (3) The present invention provides a method and device for inputting text in air, which uses a target tracking model SiamFC network to complete the tracking of the writing process of the writing stick, support the drawing of the writing trajectory of the writing stick, and realize the judgment of the end of the writing of the writing stick, thereby ensuring the real-time tracking of the air writing process, the real-time generation of the text image, and the accurate judgment of the end of the text writing;

[0082] (4) The air writing text input method and device of the present invention meet the demand for contactless human-computer interactive text input and ensure the intelligence and convenience of text input. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a main flow chart of a method for inputting text by writing in the air provided by an embodiment of the present invention;

[0084] Figure 2is a sub-flow chart of an initialization system provided by an embodiment of the present invention;

[0085] Figure 3 is a sub-flow chart for tracking a writing stick using a target tracking model provided by an embodiment of the present invention;

[0086] Figure 4 This is a sub-flow chart of writing text in the air and generating text images provided by an embodiment of the present invention;

[0087] Figure 5 It is a structural block diagram of a SiamFC network of a target tracking model provided by an embodiment of the present invention;

[0088] Figure 6 is a structural block diagram of a text recognition model provided by an embodiment of the present invention;

[0089] Figure 7 is a functional structure diagram of an air writing text input device provided by an embodiment of the present invention;

[0090] Figure 8 is a schematic diagram of a scenario of an air writing text input device provided by an embodiment of the present invention;

[0091] Fig. 9 It is a schematic diagram of an operation window of an air writing text input device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0092] The specific implementation modes provided by the present invention are described in detail below with reference to the accompanying drawings.

[0093] Figure 1-Figure 4 In the flowchart, the first letter S of the label represents a step, the value 1 of the second number represents a flowchart, the third number represents the step of the main flowchart, and the fourth number represents the step of the sub-flowchart.

[0094] like Figure 1 As shown, a method for inputting text by writing in the air of the present invention comprises the following steps:

[0095] S1.1.0: Initialize the system.

[0096] The initialization system mainly completes the display layout and parameter loading of the air writing text input system.

[0097] The S1.1.0 initialization system, sub-process as follows Figure 2 As shown, the steps are as follows:

[0098] S1.1.1: Initialize the operation window and tracking frame on the display screen.

[0099] Read the layout data of the operation window from the database, and draw the operation window on the display screen using the layout data;

[0100] like Figure 8 As shown, determine that the tracking frame is located in the center of the display screen and has a square shape, set the side length of the tracking frame to 1 / 8 of the long side of the display screen, and draw the tracking frame on the display screen.

[0101] The layout of the S1.1.1 operation window on the display screen is as follows:

[0102] like Fig. 9 As shown, the operation window includes four areas: type, write, execute and display;

[0103] The type area includes three sub-areas: numbers, symbols and English, which are used to switch the input text type;

[0104] The writing area is the restricted area of ​​the writing display, and this area displays the video of the writing stick in real time through the camera;

[0105] The execution area includes two sub-areas: end and clear. The end sub-area is used to end the text input, and the clear sub-area is used to delete the currently input text.

[0106] The display area is used to display the recognition result of the airwritten text.

[0107] S1.1.2: Initialize input text type.

[0108] Read the input text type from the database and initialize the input text type parameters.

[0109] S1.1.3: Load text recognition model parameters.

[0110] The weight parameters and hyperparameters of the text recognition model that match the input text type parameters are read from the database, and the parameters are loaded into the text recognition model.

[0111] There are three groups of weight parameters and hyperparameters for the text recognition model, corresponding to the three input text types of numbers, symbols and English. The text recognition model is trained offline using data sets for each input text type to obtain three different groups of weight parameters and hyperparameters. The three groups of weight parameters and hyperparameters are matched with the input text type parameters and saved in the database.

[0112] The S1.1.3 text recognition model uses a newly designed network, which is described as follows:

[0113] like Figure 6As shown in the figure, the text recognition model includes a text image, a feature extraction network, a classification network and an output category. The text image is the input of the model. The text image contains a complete image and a binary image to be recognized. The feature extraction network extracts the deep features of the text image. The feature extraction network includes convolution 1a and 1b, Bi-CNN1, average pooling 1, Bi-CNN2 and average pooling 2. Convolution 1a and 1b perform channel expansion on the complete image and binary image of the text image respectively. Bi-CNN1 and Bi-CNN2 are both composed of parallel conventional convolution and void convolution followed by channel splicing. Bi-CNN1 performs channel expansion on the binary image of the text image. The channel expansion map of the value map is subjected to conventional convolution. Bi-CNN1 performs dilated convolution on the channel expansion map of the complete text image map. Bi-CNN1 performs channel splicing on the two feature maps after conventional convolution and dilated convolution. Bi-CNN2 performs conventional convolution and dilated convolution on the feature map of the previous layer respectively. Bi-CNN2 performs channel splicing on the two feature maps after conventional convolution and dilated convolution. Both average pooling 1 and 2 compress the size of the feature map after channel splicing. The classification network is used for text classification and recognition. The classification network consists of fully connected 1 and fully connected 2. The output category is the result of text image recognition.

[0114] When the input text type is a number, the output category is 0-9 and the number of categories is 10;

[0115] When the input text type is symbol, the output category is commonly used symbols, and the number of categories is 38;

[0116] When the input text type is English, the output categories are uppercase and lowercase English characters, and the number of categories is 52.

[0117] The structure of the feature extraction network and classification network is shown in the following table:

[0118]

[0119] The complete image and binary image of the text image are processed by convolution 1a and 1b respectively, and the channels are expanded to 8. After two Bi-CNN and two average pooling processes, the number of channels is expanded to 32 and the size is compressed to 7×7. Then, the output category is obtained after processing by two fully connected layers. The two Bi-CNNs use parallel 3×3 regular convolution kernels and 3×3 void convolution kernels, and the two feature maps generated are fused into one feature map through channel splicing.

[0120] The input of the conventional convolution in the Bi-CNN1 layer is the binary image of the text image after channel expansion. The text information of the binary image of the text image may be incomplete but does not contain background information. The conventional convolution of the Bi-CNN1 layer can extract the local detail features of the binary image of the text image from a small receptive field and is not affected by the background information.

[0121] The input of the dilated convolution in the Bi-CNN1 layer is the complete text image after channel expansion. The text information of the complete text image is complete but contains background information. The dilated convolution in the Bi-CNN1 layer can extract the global contour features of the complete text image from a large receptive field, and supplement the text information that may be lost in the binary image while being as unaffected by background information as possible.

[0122] In the Bi-CNN2 layer, conventional convolution and dilated convolution extract feature information from different receptive fields to achieve mutual complementation of feature information under different receptive fields.

[0123] The text recognition model realizes the recognition of three types of written text: numbers, symbols and English.

[0124] The text recognition model adopts a new dual-image parallel input network structure, which integrates the global features of the complete text image and the local features of the binary text image, reduces the impact of background information on the feature extraction network and improves the accuracy and reliability of text recognition.

[0125] The offline training process of the text recognition model is as follows:

[0126] The dataset for numbers uses MNIST, the dataset for symbols uses 5,000 handwritten symbol images, and the dataset for English uses the English part of char74K. The dataset is divided into a training set and a validation set in a ratio of 9:1.

[0127] The gradient descent iteration method uses SGD, the number of iterations is 60, the number of samples per iteration is 50, the batch size is 8, and the learning rate is 10 -3 -10 -5 The intervals are selected in batches incrementally.

[0128] The images in the training set are sent to the text recognition model in sequence, the text recognition model is iteratively trained, and cross-validated through the validation set to obtain the optimal weight parameters and hyperparameters of the text recognition model.

[0129] Iterative training is performed on the three sets of data sets to obtain three different sets of weight parameters and hyperparameters.

[0130] S1.1.4: Load target tracking model parameters.

[0131] Read the weight parameters and hyperparameters of the target tracking model from the database and load the parameters into the target tracking model.

[0132] The weight parameters and hyperparameters of the target tracking model are obtained through offline training and saved in the database.

[0133] The S1.1.4 target tracking model uses a SiamFC network, which is described as follows:

[0134] like Figure 5 As shown, the input of the target tracking model SiamFC network is the template image Z of the writing stick and the search image X. The backbone of the target tracking model SiamFC network includes a feature extraction network and a response map. The two branches of the feature extraction network share weights. The left branch extracts the feature map of the template image Z, and the right branch extracts the feature map of the search image X. The feature map of the template image Z and the feature map of the search image X are convolved to generate a response map P.

[0135] The feature extraction network uses AlexNet, and the network structure of AlexNet is shown in the following table:

[0136]

[0137] The template image Z has 3 channels and a size of 127×127, the search image X has 3 channels and a size of 255×255, and the template image Z and the search image X are processed by 5 convolutions and 2 poolings respectively to obtain feature maps. The feature map of the template image Z has 256 channels and a size of 6×6, and the feature map of the search image X has 256 channels and a size of 22×22.

[0138] The object tracking model SiamFC network is used to determine the position of the current tracking box to track the writing stick.

[0139] The target tracking model SiamFC network is used to support the drawing of the moving line segment trajectory of the tracking box and generate text images.

[0140] The target tracking model SiamFC network is used to determine the maximum value of the writing stick response map to determine whether to end the text writing.

[0141] The target tracking model SiamFC network ensures the real-time tracking of the air writing process, the real-time generation of text images and the accurate judgment of the end of text writing.

[0142] The calculation formula of the response map P is as follows:

[0143]

[0144] is the bias value, is the feature map of the template image Z, is the feature map of the search image X, and * is the convolution operation.

[0145] The offline training process of the target tracking model SiamFC network is as follows:

[0146] The training data set uses GOT10K, the gradient descent iteration method uses SGD, the number of iterations is 50, the number of samples per iteration is 50,000, the batch size is 8, and the learning rate is 10 -2 -10 -5 The intervals are selected in batches incrementally.

[0147] The images in the training set are sent to the target recognition model in turn, the target tracking model SiamFC network is iteratively trained, and cross-validated through the validation set to obtain the optimal weight parameters and hyperparameters of the target tracking model SiamFC network.

[0148] S1.2.0: Move the writing stick to the tracking box.

[0149] The operator holds a writing stick and enters the aerial area monitored by the camera and moves the writing stick so that the writing stick displayed on the display screen enters the tracking frame. The writing stick can be cut from disposable hard cardboard.

[0150] S1.3.0: Track the writing stick using the target tracking model.

[0151] The real-time tracking of the writing stick is performed using the target tracking model SiamFC network described in S1.1.4 to determine the position of the current tracking box.

[0152] The S1.3.0 described above utilizes the target tracking model to track the writing stick, and the sub-process is as follows Figure 3 As shown, the steps are as follows:

[0153] S1.3.1: Read the writing stick template image.

[0154] Read the writing stick template image Z from the database.

[0155] The writing stick template image Z is created offline and saved in the database.

[0156] The process of establishing the writing stick template image Z is as follows:

[0157] Collect writing stick video through camera;

[0158] Extract a square image containing the writing stick from the video;

[0159] Resize the square image to 127×127 using traditional bilinear interpolation method;

[0160] The adjusted image is determined to be the writing stick template image Z.

[0161] S1.3.2: Extract the writing stick search image.

[0162] According to the position of the tracking frame, the image contained in the tracking frame on the display screen is intercepted, and the intercepted image is adjusted to 255×255 by a traditional bilinear interpolation method, and the adjusted image is determined as the writing stick search image X.

[0163] S1.3.3: Call the target tracking model SiamFC network to calculate the writing stick response map.

[0164] The writing stick template image Z and the writing stick search image X are sent to the target tracking model SiamFC network with loaded parameters, and the writing stick response map P is obtained after calculation by the target tracking model SiamFC network.

[0165] S1.3.4: Determine the position of the current tracking box.

[0166] The position of the current tracking frame is determined according to the writing stick response graph P, and the current tracking frame is used to track the real-time position of the writing stick.

[0167] The process of determining the position of the current tracking frame is as follows:

[0168] Search for the coordinates of the maximum value point in the writing stick response graph P;

[0169] Calculate the deviation between the coordinates of the maximum point and the coordinates of the center point of the writing stick response graph P;

[0170] Adjust the center coordinates of the tracking box according to the deviation;

[0171] Determine the adjusted tracking frame as the current tracking frame.

[0172] S1.4.0: Whether to switch.

[0173] Determine whether to switch the input text type. If yes, go to S1.5.0, if not, go to S1.6.0.

[0174] The process of S1.4.0 determining whether to switch the text type is as follows:

[0175] According to the position of the current tracking frame, it is determined whether the writing stick displayed on the display screen is in the type area of ​​the operation window. If so, the input text type needs to be switched, otherwise, the input text type does not need to be switched.

[0176] S1.5.0: Switch input text type.

[0177] The process of switching input text type in S1.5.0 is as follows:

[0178] According to the position of the current tracking frame, determining that the writing stick displayed on the display screen is located in one of the three sub-areas of numbers, symbols and English;

[0179] Set the new input text as the type of this sub-area, and save the new input text type parameter in the database;

[0180] As described in S1.1.3, load the text recognition model parameters, that is, read the weight parameters and hyperparameters of the text recognition model that match the new input text type parameters from the database, and load the parameters into the text recognition model.

[0181] S1.6.0: Write text in the air and generate text images.

[0182] Use the writing stick to write text in the air and generate text images.

[0183] The S1.6.0 described above writes text in the air and generates a text image. The sub-process is as follows Figure 4 As shown, the steps are as follows:

[0184] S1.6.1: Backup tracking frame position.

[0185] Back up the center coordinates of the tracking box.

[0186] S1.6.2: Move the writing stick.

[0187] In the space area corresponding to the writing area of ​​the operation window, the operator moves the writing stick and starts writing text.

[0188] S1.6.3: Call the target tracking model SiamFC network to determine the moving trajectory of the tracking box.

[0189] As described in S1.3.0, the target tracking model SiamFC network is used to track the writing stick, and the line segment of the current tracking frame moving relative to the center coordinates of the backup tracking frame is calculated, and this line segment is determined as the tracking frame movement trajectory.

[0190] S1.6.4: Draw the movement trajectory of the tracking box.

[0191] Draw the line segment trajectory of the tracking box movement directly in the writing area of ​​the operation window.

[0192] S1.6.5: Repeat the drawing process to track the entire movement trajectory of the box.

[0193] Repeat S1.6.1-S1.6.4 to draw the trajectory of all moving segments of the tracking box during the writing process.

[0194] S1.6.6: Cover the writing stick with your finger.

[0195] The operator covers the writing stick with one finger holding the writing stick.

[0196] S1.6.7: End the text writing operation.

[0197] As described in S1.6.3, the target tracking model SiamFC network is called to extract the maximum value in the writing stick response map P in the current target tracking model SiamFC network, and the text writing operation is terminated when the maximum value is less than the threshold.

[0198] When a finger blocks the writing stick, the similarity between the writing stick search image X and the writing stick template image Z is reduced, thereby reducing the maximum value in the writing stick response map P.

[0199] The threshold is set to the average of the maximum value in the writing stick response graph P before occlusion and the maximum value in the writing stick response graph P after occlusion.

[0200] S1.6.8: Generate text images.

[0201] The line segment trajectory drawn in the writing area of ​​the operation window is extracted to generate a text image, including a complete image and a binary image.

[0202] The process of generating text images is as follows:

[0203] Capture an image of the writing area of ​​the operation window from the display screen;

[0204] Use the traditional OTSU algorithm to segment the captured image and obtain the segmentation threshold;

[0205] Use the segmentation threshold to binarize the captured image into an image with foreground as 0 and background as 1;

[0206] Fill the background of the binary image and adjust it to a square image 1;

[0207] The square image 1 is reduced to 28×28 by bilinear interpolation method;

[0208] The square image 1 with a channel of 1 and a size of 28×28 is determined as a binary image of the text image. The binary image does not contain background information but the text information may be incomplete. The loss of text information is caused by the segmentation algorithm.

[0209] Fill the captured image with background and adjust it to a square image 2;

[0210] The square image 2 is reduced to 28×28 by bilinear interpolation method;

[0211] The square image 2 with 3 channels and a size of 28×28 is determined to be a complete text image. The complete image contains background information but the text information is complete.

[0212] S1.7.0: Use the text recognition model to recognize text images.

[0213] The complete image and the binary image of the text image are sent to the text recognition model described in S1.1.3 or S1.5.0. After calculation by the text recognition model, an output category matching the input text type described in S1.1.3 is obtained. The output category is the recognized new text;

[0214] The complete text image and the binary image can complement each other's feature information through feature extraction and channel splicing of the Bi-CNN layer in the text recognition model.

[0215] S1.8.0: Added text display.

[0216] The process of adding text display in S1.8.0 is as follows:

[0217] Read the text string parameter displayed in the operation window display area from the database;

[0218] Add the recognized new text characters to the text string parameter and save the new text string parameter to the database;

[0219] Output the new text string parameter to the operation window display area of ​​the display screen for display.

[0220] S1.9.0: Whether to clear.

[0221] Determine whether to clear the added displayed text, if so go to S1.10.0, if not go to S1.11.0.

[0222] The process of S1.9.0 determining whether to clear the added displayed text is as follows:

[0223] As described in S1.6.3, call the target tracking model SiamFC network to determine the position of the current tracking box;

[0224] According to the position of the current tracking frame, it is determined whether the writing stick displayed on the display screen is in the clearing sub-area position of the operation window execution area. If so, the additional displayed text needs to be cleared, otherwise it does not need to be cleared.

[0225] S1.10.0: Clear the text added to the display.

[0226] The process of clearing the added displayed text in S1.10.0 is as follows:

[0227] Read the text string parameter displayed in the operation window display area from the database;

[0228] Delete the last character in the text string parameter and save the new text string parameter to the database;

[0229] The new string parameter is output to the operation window display area of ​​the display screen for display.

[0230] S1.11.0: Is it finished?

[0231] Determine whether the input of all characters in the text is finished. If so, end it. If not, go to S1.4.0.

[0232] The process of S1.11.0 for determining whether the input of all characters in the text is completed is as follows:

[0233] According to S1.6.3, call the target tracking model SiamFC network to determine the position of the current tracking box;

[0234] According to the position of the current tracking frame, it is determined whether the writing stick displayed on the display screen is in the end sub-area position of the operation window execution area. If so, the text input is terminated, and if not, the text input can continue.

[0235] An air writing text input device, the functional structure is as follows Figure 7 As shown, it includes the following modules:

[0236] The camera module 10, composed of the sensor HD98, is used to collect real-time video of the writing and operation of the writing stick in the air, including:

[0237] Monitor the aerial area directly ahead in real time and collect information about this area in the form of video;

[0238] Collect the writing action of the writing stick in the air area of ​​the writing area of ​​the operation window;

[0239] Collect the operation of the writing stick on the numbers, symbols, and English sub-areas in the operation window type area;

[0240] The collection writing stick clears the operation window execution area and ends the operation of the sub-area air area.

[0241] The tracking module 20 includes a target tracking model, which is used to track the writing stick, generate a text image, and end the text writing, including:

[0242] Offline training of the target tracking model to obtain the optimal weight parameters and hyperparameters;

[0243] Load the weight parameters and hyperparameters of the target tracking model;

[0244] The position of the current tracking frame is determined by the target tracking model, and the writing stick is tracked;

[0245] The target tracking model is used to calculate the line segment trajectory of the tracking frame, and the text image is generated after all the line segment trajectories are drawn;

[0246] The target tracking model is used to determine the maximum value of the writing stick response graph to determine whether to end the text writing.

[0247] The recognition module 30 includes a text recognition model for recognizing three types of text, namely, numbers, symbols and English written in the air, including:

[0248] Offline training of text recognition models for numbers, symbols, and English to obtain three sets of optimal weight parameters and hyperparameters;

[0249] Load the weight parameters and hyperparameters of the text recognition model for numbers, symbols, and English;

[0250] The text image written in the air is recognized through the text recognition model, and the output category is the recognized new text.

[0251] The display module 40 is composed of an 11-inch liquid crystal display screen, which is used to display the operation window, video, tracking frame, text image and recognized text, including:

[0252] Display the layout of the operation window to provide the operator with a visual operation area;

[0253] Display the video of writing and operation of the writing stick in the air collected by the camera module;

[0254] Displaying a tracking frame that determines the current position using a tracking module;

[0255] Display text images generated using the tracking model;

[0256] Displays new text recognized by the recognition module.

[0257] The database module 50, which is composed of a solid-state storage hard disk in the embedded module, is used to store data and parameters of the air writing text input method, including:

[0258] Save the layout data of the operation window;

[0259] Save the input text type parameter;

[0260] Save the weight parameters and hyperparameters of the target tracking model;

[0261] Save the weight parameters and hyperparameters of the text recognition model for numbers, symbols, and English;

[0262] Save the writing board template image Z;

[0263] Save the text string parameters displayed in the operation window display area;

[0264] Caches auxiliary data for the airwrite text input method.

[0265] Embedded module 60: composed of a carrier board Jetson Xavier NX of a solid state storage hard disk, used for driving an air writing text input device and running a program of an air writing text input method, including:

[0266] Connect to the camera, drive the camera, and run the camera module;

[0267] Connected to the display screen, drives the display screen, and runs the display module;

[0268] Running the database module to store the program, data and parameters of the writing text input method through the solid-state storage hard disk;

[0269] Run the tracking module and recognition module to complete the tracking and recognition functions of the air-written text input;

[0270] Provide GPU computing power for the models of the tracking module and the recognition module.

[0271] like Figure 8 As shown, the writing stick is cut from disposable hard cardboard, the display screen adopts an 11-inch LCD display, the camera adopts HD98 of GUC, the embedded platform adopts Jetson Xavier NX of NVIDIA, the tracking frame is a square frame that is always displayed on the display screen, the camera is installed directly above the display screen, the camera is connected to the embedded platform through a USB interface, the display screen is connected to the embedded platform through an HDMI interface, the writing stick is written in the space directly in front of the camera in a handheld manner, the camera collects videos of writing and operation of the writing stick, and the embedded platform is used to drive the air writing text input device and run the air writing text input method program.

[0272] The above embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given, but the protection scope of the present invention is not limited to the above embodiments.

Claims

1. A method for inputting text by writing in the air, characterized in that: The steps include: Step 1: Initialize the air writing text input system; Step 2: Hold the writing stick and enter the aerial area monitored by the camera, and move the writing stick into the tracking frame; Step 3: Use the target tracking model to track the writing stick and determine the position of the current tracking box; Step 4: Determine whether to switch the input text type, if yes, go to step 5, if not, go to step 6; Step 5: Switch the input text type; Step 6: Use the writing stick to write text in the air and generate a text image; Step 7: Use the text recognition model to recognize the text image and identify the text; Step 8: Add text display; Step 9: Determine whether to clear the added displayed text, if yes, go to step 10, if not, go to step 11; Step 10: Clear the added text; Step 11: Determine whether the input of all characters in the text is completed, if so, then end, if not, go to step 4.

2. The method for inputting text by writing in the air according to claim 1, characterized in that: In step 1, initializing the air writing text input system includes the following steps: Step 1-1: Initialize the operation window and tracking frame on the display screen; Step 1-2: Initialize the input text type; Step 1-3: Load the text recognition model parameters that match the input text type; Step 1-4: Load the target tracking model parameters.

3. The method for inputting text in the air according to claim 2, characterized in that: In the steps 1-3, the text recognition model uses a newly designed network, which specifically includes the following: The text recognition model includes text images, feature extraction networks, classification networks and output categories. The text image is the input of the model. The text image contains the complete image and binary image to be recognized. The feature extraction network extracts the deep features of the text image. The feature extraction network includes convolution layer, Bi-CNN layer, average pooling layer, Bi-CNN layer and average pooling layer. The convolution layer performs channel expansion on the complete image and binary image of the text image respectively. The Bi-CNN layer consists of parallel conventional convolution and void convolution followed by channel splicing. The first Bi-CNN layer performs conventional convolution on the channel feature map of the binary image of the text image. The first Bi-CNN layer performs dilated convolution on the channel expansion map of the complete text image. The first Bi-CNN layer performs channel splicing on the two feature maps after regular convolution and dilated convolution. The second Bi-CNN layer performs regular convolution and dilated convolution on the feature map of the previous layer respectively. The second Bi-CNN layer performs channel splicing on the two feature maps after regular convolution and dilated convolution. Both average pooling layers perform size compression on the feature maps after channel splicing. The classification network is used for text classification and recognition. The classification network consists of two fully connected layers, and the output category is the result of text image recognition. When the input text type is a number, the output category of the text recognition model is 0-9, and the number of categories is 10; When the input text type is a symbol, the output category of the text recognition model is the commonly used symbols, and the number of categories is 38; When the input text type is English, the output categories of the text recognition model are uppercase and lowercase English characters, and the number of categories is 52; The text recognition model adopts a new dual-image parallel input network structure. The text recognition model recognizes three types of written text: numbers, symbols and English. The text recognition model integrates the global features of the complete text image and the local features of the binary text image. The text recognition model reduces the impact of background information on the feature extraction network.

4. The method for inputting text by writing in the air according to claim 1, characterized in that: In step 3, tracking the writing stick using the target tracking model includes the following steps: Step 3-1: Read the writing stick template image; Step 3-2: extracting the writing stick search image according to the position of the tracking frame; Step 3-3: Call the target tracking model SiamFC network to calculate the response map between the template image of the writing stick and the search image; Step 3-4: Determine the position of the current tracking frame according to the writing stick response graph.

5. The method for inputting text by writing in the air according to claim 1, characterized in that: In step 4, the process of determining whether to switch the text type is as follows: According to the position of the current tracking frame, it is determined whether the writing stick is in the position of the operation window type area. If it is, the input text type needs to be switched. Otherwise, the input text type does not need to be switched.

6. The method for inputting text by writing in the air according to claim 1, characterized in that: In step 5, the process of switching the input text type is as follows: According to the position of the current tracking frame, determine that the writing stick is located in one of the three sub-areas of numbers, symbols and English; Set the new input text to the type of this subarea; Load the text recognition model parameters that match the new input text type.

7. The method for inputting text by writing in the air according to claim 1, characterized in that: In step 6, writing text in the air and generating a text image includes the following steps: Step 6-1: Backup the position of the tracking frame; Step 6-2: Move the writing stick in the air area corresponding to the writing area of ​​the operation window; Step 6-3: Call the target tracking model SiamFC network to determine the trajectory of the current tracking frame relative to the backup tracking frame; Step 6-4: Draw the trajectory of the tracking frame directly in the writing area of ​​the operation window; Step 6-5: Repeat the above steps to draw the entire moving trajectory of the tracking box during the writing process; Step 6-6: Cover the writing stick with your finger; Step 6-7: extract the maximum value of the response map in the SiamFC network of the current target tracking model, and end the text writing operation when the maximum value of the response map is less than the threshold; Step 6-8: Extract the trajectory drawn in the writing area of ​​the operation window and generate a text image, including a complete image and a binary image.

8. The method for inputting text by writing in the air according to claim 1, characterized in that: The process of determining whether to clear the added displayed text in step 9 is as follows: Call the target tracking model SiamFC network to determine the position of the current tracking box; According to the position of the current tracking frame, it is determined whether the writing stick is in the clearing sub-area of ​​the operation window execution area. If so, the added displayed text needs to be cleared, otherwise, it does not need to be cleared.

9. The method for inputting text by writing in the air according to claim 1, characterized in that: The process of determining whether the input of all characters in the text is completed in step 11 is as follows: Call the target tracking model SiamFC network to determine the position of the current tracking box; According to the position of the current tracking frame, it is determined whether the writing stick is in the end sub-area of ​​the operation window execution area. If so, the text input is terminated, and if not, the text input can be continued.

10. An air writing text input device using the air writing text input method according to claim 1, comprising the following modules: Camera module: used to collect real-time video of writing and operation of the writing stick in the air; Tracking module: contains the target tracking model, which is used to track the writing stick, generate text images and end text writing; Recognition module: contains a text recognition model, which is used to recognize three types of texts: numbers, symbols and English written in the air; Display module: used to display the operation window, video, tracking frame, text image and recognized text; Database module: used to store data and parameters of the air writing text input method; Embedded module: used for driving the air writing text input device and running the air writing text input method program; The embedded module is connected to the camera and the display screen; The embedded module runs the database module to store the program, data and parameters of the writing text input method; The embedded module runs the tracking module and the recognition module to complete the tracking and recognition functions of the air-written text input; the embedded module provides GPU computing power for the models of the tracking module and the recognition module.

Images