Air handwritten text acquisition method based on improved yolov5 network and monocular vision

Abstract

The application relates to the technical field of intelligent human-computer interaction, in particular to an aerial handwritten text collection method based on an improved yolov5 network and monocular vision, which comprises the following steps: constructing an initial fingertip dataset and performing pretreatment to obtain a target fingertip dataset, improving the yolov5 network based on an efficient pyramid segmentation attention module EPSA and a weighted bidirectional feature pyramid network BiFPN, obtaining an improved yolov5 network, inputting a training set of the target fingertip dataset into the improved yolov5 network to obtain a fingertip detection model, obtaining real-time two-dimensional video images of aerial handwritten text based on a monocular camera and inputting the fingertip detection model, and forming the aerial handwritten text based on a coordinate system virtual sliding technology, so that the technical problems that, in the prior art, a 3D sensor is large and expensive, resulting in insufficient universality, the requirement for a WIFI environment is relatively strict, resulting in relatively large limitation, and a fingertip is a small target, resulting in low detection accuracy and high omission rate when the yolov5 network detects the fingertip are solved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of intelligent human-computer interaction, and in particular to an air handwriting text collection method based on an improved yolov5 network and monocular vision. BACKGROUND

[0002] As one of main carriers of information, characters play a very important role in the field of human-computer interaction. As a new type of human-computer interaction method, the air handwriting interaction method is more natural and humanized in writing, can enable users to perform non-contact human-computer interaction, has less limitations, and provides a more comfortable and free experience for users. One aspect of the air handwriting interaction method is the collection of air handwriting text.

[0003] At present, the collection of air handwriting text still has some problems. On the one hand, the recognition of finger tips in the collection of air handwriting text is mainly based on the yolov5 network to detect and locate the finger tips. However, since the finger tips are small targets, the yolov5 network has the technical problems of low detection accuracy and high missing rate when detecting the finger tips. On the other hand, the collection of air handwriting text is mainly based on 3D sensors (such as Leap Motion, Kinect, etc.) with depth information and WIFI signals. However, the collection of air handwriting text based on 3D sensors with depth information has poor universality due to the large volume and high price of the 3D sensors. The collection of air handwriting text based on WIFI signals has great limitations due to strict requirements for WIFI environment. SUMMARY

[0004] The application aims to provide an air handwriting text collection method based on an improved yolov5 network and monocular vision, to solve the technical problems of poor universality due to the large volume and high price of 3D sensors, great limitations due to strict requirements for WIFI environment, and low detection accuracy and high missing rate of the yolov5 network when detecting finger tips.

[0005] To achieve the above-mentioned purpose, the application provides the following technical scheme.

[0006] An air handwriting text collection method based on an improved yolov5 network and monocular vision, the method comprising the following steps:

[0007] Obtaining a plurality of finger tip pictures containing finger tips, and constructing an initial finger tip dataset based on the plurality of finger tip pictures;

[0008] Preprocess the initial fingertip dataset, obtain the target fingertip dataset, and divide the target fingertip dataset into a training set and a test set in proportion;

[0009] The yolov5 network is improved based on the efficient pyramid segmentation attention module EPSA and the weighted bidirectional feature pyramid network BiFPN, and the improved yolov5 network is obtained, wherein the improvement point is that the residual module bottleneck in the C3 module is replaced by the efficient pyramid segmentation attention module EPSA to obtain the C3EPSA module, the deepest C3 module in the backbone network and the neck network of the yolov5 network is replaced by the C3EPSA module to enhance the feature extraction capability of the yolov5 network, and the path aggregation network PANet in the yolov5 network is replaced by the weighted bidirectional feature pyramid network BiFPN to better fuse the feature maps of different levels obtained by the backbone network.

[0010] The training set is input into the improved yolov5 network for training and output of a prediction value, a loss value is calculated by using a loss function, and when the loss value stops decreasing, a fingertip detection model is obtained.

[0011] Based on the monocular camera, the trajectory of the fingertip movement is collected, the real-time two-dimensional video image of the air handwritten text is obtained, and each frame of the real-time two-dimensional video image of the air handwritten text is input into the fingertip detection model in turn to obtain a plurality of target boxes containing fingertips.

[0012] Based on the information of the plurality of target boxes containing fingertips and the coordinate system virtual sliding technology, a fingertip coordinate sequence composed of fingertip coordinates of each frame of the real-time two-dimensional video image of the air handwritten text is obtained, each fingertip coordinate in the fingertip coordinate sequence is connected to form the air handwritten text.

[0013] In the air handwritten text acquisition method based on the improved yolov5 network and monocular vision provided in the application, part of the fingertip pictures are obtained through the network, and the other part of the fingertip pictures are obtained by shooting different people's fingers in different backgrounds, different light conditions, different angles and different distances through the monocular camera.

[0014] In the air handwritten text acquisition method based on the improved yolov5 network and monocular vision provided in the application, the preprocessing of the initial fingertip dataset, the obtaining of the target fingertip dataset, and the dividing of the target fingertip dataset into a training set and a test set in proportion include the following steps:

[0015] The target detection dataset labeling tool lableimg is used to label a plurality of fingertip pictures in the initial fingertip dataset, and a rectangular frame is used to label the position of the fingertip in the picture.

[0016] The initial fingertip dataset after annotation is expanded using mosaic stitching, affine transformation, and segmentation and padding data augmentation methods to obtain the target fingertip dataset.

[0017] The target fingertip dataset was divided into training and testing sets in an 8:2 ratio.

[0018] In the aerial handwritten text acquisition method based on the improved YOLOv5 network and monocular vision provided by this invention, the specific operation mode of the efficient pyramid segmentation attention module EPSA is as follows:

[0019] The first input feature map X of the efficient pyramid segmentation attention module EPSA is passed through a Conv1×1 module to obtain the second input feature map X1;

[0020] For the second input feature map X1, the second input feature map X1 is divided into S segmentation feature maps X'1 on average according to the number of channels of the second input feature map X1;

[0021] For S segmented feature maps X'1, feature maps F at different scales are extracted from the first input feature map X using group convolution with S different convolution kernels. i Where i = 0, 1, ..., S-1;

[0022] Feature maps F at different scales i The input is fed into the SEWeight module for learning, extracting feature maps F at different scales. i Channel attention is used to obtain feature maps F at each different scale. i Channel attention vector Z i Where i = 0, 1, ..., S-1;

[0023] Using the softmax function to process feature maps F at different scales i Channel attention vector Z i Perform weight recalibration to obtain the weight vector Y. i Where i = 0, 1, ..., S-1;

[0024] The feature map F at the corresponding scale i With weight vector Y i Perform Hadamard product operations to obtain feature maps O at different scales after channel attention weighting. i S feature maps O i Perform splicing and fusion to obtain multi-scale channel attention-weighted feature maps O, where i = 0, 1, ..., S-1;

[0025] The feature map O after the multi-scale channel attention weighting is input into a Conv1x1 module and is summed with the first input feature map X to obtain an output feature map X' with multi-scale spatial information and cross-channel attention.

[0026] In the method for collecting aerial handwritten text based on the improved yolov5 network and monocular vision provided in the embodiments of the application, when the improved yolov5 network is trained by using a training set, the network training parameters are set as follows:

[0027] The initial network weight is yolov5s.pt, the training size is set to 640*640, the training iteration number is set to 50, the batch size batch-size is set to 16, and the initial learning rate is 0.01.

[0028] In the method for collecting aerial handwritten text based on the improved yolov5 network and monocular vision provided in the application, the loss function calculation includes rectangular frame loss calculation, confidence loss calculation and classification loss calculation.

[0029] The rectangular frame loss calculation calculates the rectangular frame loss by using a CIoU loss function, and the calculation formula is as follows:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] Wherein, S1 is the intersection area of the real frame and the predicted frame, S2 is the union area of the real frame and the predicted frame, ω t is the width of the real frame, h t is the height of the real frame, ω p is the width of the predicted frame, h p is the height of the predicted frame, a is the center point of the real frame, b is the center point of the predicted frame, δ 2 (a, b) is the Euclidean distance between a point and b point, λ is a weight coefficient, μ is a correlation coefficient of the length-width ratio of the real frame and the length-width ratio of the predicted frame, L CIoU is the rectangular frame loss of a single predicted frame.

[0036] The confidence loss calculation calculates the confidence loss by using a binary cross-entropy loss function, and the calculation formula is as follows:

[0037] L conf= -vlnm - (1-v)ln(1-m) ;

[0038] wherein, v is a confidence label, m is a predicted confidence, L conf is the confidence loss of a single prediction box;

[0039] The classification loss calculation calculates the classification loss by using a binary cross-entropy loss function, and the calculation formula is as follows:

[0040] L class = -εlnp - (1-ε)ln(1-p) ;

[0041] wherein, ε is a label probability, p is a predicted probability, L class is the classification loss of a single prediction box;

[0042] The calculation formula of the total loss L of the loss function is as follows:

[0043]

[0044] wherein, ω1, v2 and ω3 are weights of the rectangular frame loss, the classification loss and the confidence loss in the total loss L respectively, n is the number of positive samples, m is the number of negative samples, and e is the weight of the confidence loss of the positive sample in all confidence losses.

[0045] In the air handwritten text acquisition method based on the improved yolov5 network and monocular vision provided in the application, after the fingertip detection model is obtained, the fingertip detection model needs to be evaluated and tested, including the following steps:

[0046] The fingertip detection model is evaluated by using a model evaluation index, and the model evaluation index includes precision P, recall rate R and average precision mean mAP, and the formula is as follows:

[0047]

[0048]

[0049]

[0050]

[0051] wherein, T is the number of correct prediction of positive samples, F is the number of false prediction of positive samples, N is the number of false prediction of negative samples, P(R) is a function of the relationship between the precision P and the recall rate R, AP is the average precision, AP k APk represents the average precision of the kth target, and n represents the number of target categories.

[0052] The test set is input into the fingertip detection model for testing.

[0053] In the improved yolov5 network-based and monocular vision-based aerial handwritten text acquisition method provided by the application, the information of the target frame containing the fingertips and the coordinate system virtual sliding technology are used to obtain a fingertip coordinate sequence of the fingertips of the real-time two-dimensional video image of each frame of aerial handwritten text, and each fingertip coordinate in the fingertip coordinate sequence is connected to form the aerial handwritten text, including the following steps:

[0054] Based on the left upper vertex position coordinates of the target frame containing the fingertips, the width of the target frame and the height of the target frame, coordinate transformation is performed to obtain the fingertip coordinates of the real-time two-dimensional video image of each initial frame of aerial handwritten text;

[0055] Based on the fingertip coordinates of the real-time two-dimensional video image of each initial frame of aerial handwritten text and the coordinate system virtual sliding technology, the fingertip coordinate sequence of the real-time two-dimensional video image of the aerial handwritten text is obtained, and the coordinate system virtual sliding technology is:

[0056] When the fingertip coordinates of the real-time two-dimensional video image of the current frame of aerial handwritten text are obtained, the horizontal coordinates of the fingertip coordinates of the real-time two-dimensional video image of each frame of aerial handwritten text before the current frame need to be translated to the left, and the vertical coordinates remain unchanged.

[0057] The fingertip coordinates of the real-time two-dimensional video image of the aerial handwritten text are sequentially connected to form the aerial handwritten text.

[0058] Compared with the prior art, the application has the following beneficial effects:

[0059] The improved yolov5 network is obtained by improving the yolov5 network by using an efficient pyramid segmentation attention module EPSA and a weighted bidirectional feature pyramid network BiFPN, an initial fingertip dataset is constructed by using a plurality of fingertip pictures containing fingertips, a target fingertip dataset is obtained by preprocessing the initial fingertip dataset, the target fingertip dataset is divided into a training set and a test set, a fingertip detection model is obtained by inputting the training set into the improved yolov5 network for training, a monocular camera which is cheap, small in size and good in versatility is used to collect the track of the fingertip movement, a real-time two-dimensional video image of the air handwritten text is obtained, the real-time two-dimensional video image of each frame of the air handwritten text is sequentially input into the fingertip detection model to obtain a plurality of target boxes containing fingertips, based on the information of the plurality of target boxes containing fingertips and a coordinate system virtual sliding technology, a fingertip coordinate sequence composed of fingertip coordinates of each frame of the real-time two-dimensional video image of the air handwritten text is obtained, the fingertip coordinates of each frame of the real-time two-dimensional video image of the air handwritten text in the fingertip coordinate sequence are connected to form the air handwritten text, the air handwritten interaction method improves the detection accuracy of the fingertip and the universality of the air handwritten interaction method, reduces the missed detection rate of the fingertip detection of the air handwritten interaction method and the limitation of the air handwritten interaction method, solves the technical problems in the prior art that the 3D sensor is large in size and expensive, resulting in insufficient universality, the requirement for the WIFI environment is relatively strict, resulting in relatively large limitation, and the fingertip is a small target, resulting in low detection accuracy and high missed detection rate of the fingertip detection of the yolov5 network.

[0060] It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter.

[0061] The foregoing and other aspects, embodiments and features of the present teachings can be better understood from the following detailed description taken in conjunction with the accompanying drawings. Additional features of the present teachings will be described or will become apparent in the course of the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced as follows. The drawings are not intended to be drawn to scale, and in order to be clear, not every component will be marked in each drawing. The drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings. Among them:

[0063] Figure 1 The flow chart of the air handwritten text acquisition method based on the improved yolov5 network and monocular vision in the embodiments of the present application.

[0064] Figure 2 The architecture diagram of the yolov5 network in the embodiments of the present application.

[0065] Figure 3 The architecture diagram of the improved yolov5 network in the embodiments of the present application.

[0066] Figure 4 The architecture diagram of the efficient pyramid segmentation attention module EPSA in the embodiments of the present application.

[0067] Figure 5 The structure diagram of C3EPSA in the embodiments of the present application.

[0068] Figure 6 The schematic diagram of the fingertip picture in the embodiments of the present application.

[0069] Figure 7 The effect diagram of the conversion of the fingertip picture before and after preprocessing in the embodiments of the present application.

[0070] Figure 8 The effect diagram of the conversion of the two-dimensional video image before and after fingertip detection model detection in the embodiments of the present application.

[0071] Figure 9 The effect diagram of the air handwritten text recognition in the embodiments of the present application. DETAILED DESCRIPTION

[0072] The embodiments of the present application provide an air handwritten text acquisition method based on an improved yolov5 network and monocular vision, which is used to solve the technical problems of low detection accuracy, high missing detection rate of the yolov5 network in detecting fingertips due to the large volume and high price of 3D sensors, strict requirements on WIFI environment, and small target of fingertips.

[0073] The concept of the embodiment of the application is to improve the yolov5 network by using the efficient pyramid segmentation attention module EPSA and the weighted bidirectional feature pyramid network BiFPN to obtain an improved yolov5 network, construct an initial fingertip dataset by a plurality of fingertip pictures containing fingertips, preprocess the initial fingertip dataset to obtain a target fingertip dataset, divide the target fingertip dataset into a training set and a test set, input the training set into the improved yolov5 network to train and obtain a fingertip detection model, use a monocular camera which is cheap, small in size and good in versatility to collect the trajectory of the fingertip movement, obtain a real-time two-dimensional video image of the air handwritten text, input the real-time two-dimensional video image of each frame of the air handwritten text into the fingertip detection model in sequence to obtain a plurality of target boxes containing fingertips, based on the information of the plurality of target boxes containing fingertips and the coordinate system virtual sliding technology, obtain a fingertip coordinate sequence composed of the fingertip coordinates of each frame of the real-time two-dimensional video image of the air handwritten text, connect the fingertip coordinates of each frame of the real-time two-dimensional video image of the air handwritten text in the fingertip coordinate sequence to form the air handwritten text, and improve the detection accuracy of the fingertip by the air handwritten interaction method and the universality of the air handwritten interaction method, reduce the missed detection rate of the fingertip detection by the air handwritten interaction method and the limitation of the air handwritten interaction method.

[0074] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described below in connection with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without any inventive effort fall within the scope of protection of the present application. Unless otherwise defined, the technical terms or scientific terms used herein should have the usual meanings understood by a person having ordinary skills in the art to which the present application belongs.

[0075] The terms "first", "second", and similar terms used in the specification and claims of the present patent application do not denote any order, quantity, or importance, but are used to distinguish different components. Similarly, the singular forms "a", "an", and "the" do not denote a quantity restriction, but denote the presence of at least one, unless the context clearly indicates otherwise. The terms "comprise", "comprising", and similar terms mean that the elements or objects before "comprise" or "comprising" encompass the features, integers, steps, operations, elements, and / or components listed after "comprise" or "comprising", and do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The terms "up", "down", "left", "right", and the like only represent relative positional relationships, which may change accordingly when the absolute positions of the described objects change.

[0076] The embodiments are only specific embodiments of the present application, and the application will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0077] In combination Figures 1 to 5 As shown in the drawings, the embodiments of the present application provide an aerial handwritten text acquisition method based on an improved yolov5 network and monocular vision, which comprises the following steps:

[0078] Step S10: Obtain a plurality of fingertip pictures containing fingertips, and construct an initial fingertip dataset based on the plurality of fingertip pictures, wherein a part of the fingertip pictures are obtained through a network, and another part of the fingertip pictures are obtained by shooting different people's fingers in different backgrounds, different lighting conditions, different angles, and different distances through a monocular camera.

[0079] In order to obtain a robust fingertip detection model, a fingertip dataset with high data quality and sufficient data is required. Therefore, in the present embodiment, the number of fingertip pictures is 1600, of which 800 are obtained by downloading from a network, and the remaining 800 fingertip pictures are obtained by shooting different people's fingers in different backgrounds, different lighting conditions, different angles, and different distances through a monocular camera, so as to obtain various fingertip data. Figure 6 As shown in the drawings, it can be known that the backgrounds, lighting, and shooting angles of different fingertip pictures are different.

[0080] Step S20: Preprocess the initial fingertip dataset to obtain a target fingertip dataset, and divide the target fingertip dataset into a training set and a test set according to a proportion.

[0081] In some embodiments, the preprocessing of the initial fingertip dataset, obtaining the target fingertip dataset, and proportionally dividing the target fingertip dataset into a training set and a test set include the following steps:

[0082] Step S201: Use the target detection dataset labeling tool lableimg to label a plurality of fingertip pictures in the initial fingertip dataset, and use a rectangular frame to label the positions of the finger fingertips in the pictures.

[0083] Step S202: Based on the mosaic splicing, affine transformation, and segmentation filling data enhancement method, the initial fingertip dataset after labeling is expanded to obtain the target fingertip dataset.

[0084] Step S203: Divide the target fingertip dataset into a training set and a test set according to a ratio of 8:2.

[0085] Among them, as shown in the figure Figure 7 The figure shows the conversion effect of the fingertip picture in the initial fingertip dataset before and after the target fingertip dataset is obtained by preprocessing.

[0086] Step S30: Based on the efficient pyramid segmentation attention module EPSA and the weighted bidirectional feature pyramid network BiFPN, the yolov5 network is improved to obtain an improved yolov5 network, wherein the improvement point is that the residual module bottleneck in the C3 module is replaced by the efficient pyramid segmentation attention module EPSA to obtain the C3EPSA module, the deepest C3 module in the backbone network and the neck network of the yolov5 network is replaced by the C3EPSA module to enhance the feature extraction capability of the yolov5 network, and the path aggregation network PANet in the yolov5 network is replaced by the weighted bidirectional feature pyramid network BiFPN to better fuse the features of different levels obtained by the backbone network.

[0087] Among them, as shown in the figure Figure 2 The figure shows the architecture of the yolov5 network, and the figure Figure 3An architecture diagram of an improved yolov5 network is shown, including a backbone network Backbone, a neck network Neck and a head network Head, the backbone network Backbone is used to extract the features of the image, and the original input image is converted into a multi-layer feature map, first, the Focus module is used to obtain a two times down-sampling feature map without information loss, the two times down-sampling feature map without information loss is input into a first Conv3x3 module, in the first Conv3x3 module, local spatial information is extracted through convolution operation, the feature value distribution in the neural network is normalized through the BN layer, and the non-linear transformation ability is introduced through the activation function, so as to realize the conversion and extraction of the input two times down-sampling feature map without information loss, then, the feature map obtained through the first Conv3x3 module is input into the C3 true module, the depth and the receptive field of the network are increased, and the feature extraction capability of the network is improved, then, after alternately passing through two groups of Conv3x3 modules and C3 true modules, the feature map is again input into the Conv3x3 module and the C3EPSA module, and the feature map of different levels is obtained, finally, the feature map obtained by the C3EPSA is input into the SPPF module, and the multi-scale feature fusion is realized through cascading multiple pooling layers; the neck network Neck is responsible for multi-scale feature fusion of the feature map, in order to fuse the feature maps of different levels together, the feature map needs to be up-sampled or down-sampled, the Upsample module is used to up-sample the output feature map of the SPPF module of the backbone network Backbone, and then the up-sampled feature map is input into the BiFPN2 module together with the output feature map of the C3 true module in the second group of the two groups of Conv3x3 modules and C3 true modules Figure 1 The output feature map of the first C3 false module is up-sampled by the Upsample, and then the up-sampled feature map is input into the BiFPN2 module together with the output feature map of the C3 true module in the first group of the two groups of Conv3x3 modules and C3 true modules Figure 1 The output feature map of the second C3 false module is input into the BiFPN2 module together with the output feature map of the first C3 false module and the output feature map of the C3 true module in the second group of the two groups of Conv3x3 modules and C3 true modules Figure 1 The output feature map of the third C3 false module is input into the BiFPN3 module together with the output feature map of the SPPF module Figure 1The fused feature map is input into a C3EPSA module for feature extraction, and the feature maps obtained by the second C3 false module, the third C3 false module and the C3EPSA module are input into a Detect module, wherein the BiFPN2 module and the BiFPN3 module both belong to a BiFPN; a head network Head is responsible for final regression prediction and is composed of the Detect module and internally includes a convolution layer, a pooling layer and a full connection layer, and is responsible for multi-scale target detection on the obtained feature map to obtain target class probability, object score and boundary box position information.

[0088] In some embodiments, the specific operation mode of the efficient pyramid segmentation attention module EPSA is as follows:

[0089] The first input feature map X of the efficient pyramid segmentation attention module EPSA is input into a Conv1x1 module to obtain a second input feature map X1.

[0090] The second input feature map X1 is evenly divided into S segmentation feature maps X'1 according to the channel number of the second input feature map X1.

[0091] In this embodiment, the channel number of the second input feature map X1 is C, and S is 4, so the channel number of each segmentation feature map X'1 is

[0092] The S segmentation feature maps X'1 are input into a group convolution with S different convolution kernels to extract different scale feature maps F i , i = 0, 1,..., S-1.

[0093] In this embodiment, because S is 4, the four segmentation feature maps X'1 are input into a group convolution with four different convolution kernels to extract four different scale feature maps F i , i = 0, 1, 2, 3. The convolution kernels of the four group convolutions are 3, 5, 7 and 9, respectively, and the corresponding group numbers are 1, 4, 8 and 16. When the convolution kernel K = 3, the group number G = 1, and when the convolution kernel K > 3, the group number G and the convolution kernel K have the following relationship:

[0094]

[0095] The different scale feature maps F i are input into an SEWeight module for learning to extract the channel attention of the different scale feature maps F i , and obtain the channel attention vector Z i of each different scale feature map F i, where i = 0, 1, ..., S-1.

[0096] The following formula can be used to calculate it:

[0097] Z i =SEWeight(F i ), i = 0, 1, 2, 3.

[0098] Using the softmax function to process feature maps F at different scales i Channel attention vector Z i Perform weight recalibration to obtain the weight vector Y. i , where i = 0, 1, ..., S-1.

[0099] The following formula can be used to calculate it:

[0100] Y i =softmax(Z) i ), i = 0, 1, 2, 3.

[0101] The feature map F at the corresponding scale i With weight vector Y i Perform Hadamard product operations to obtain feature maps O at different scales after channel attention weighting. i S feature maps O i Perform splicing and fusion to obtain multi-scale channel attention-weighted feature maps O, where i = 0, 1, ..., S-1.

[0102] The following formula can be used to calculate it:

[0103] O i =F i ⊙Y i i = 0, 1, 2, 3

[0104] O = Cat([O0,O1,O2,O4])

[0105] The feature map O after multi-scale channel attention weighting is input into the Conv1×1 module and then summed with the first input feature map X to obtain the output feature map X′ which has multi-scale spatial information and cross-channel attention.

[0106] Step S40: Input the training set into the improved YOLOv5 network for training and output the predicted value. Calculate the loss value using the loss function. When the loss value stops decreasing, obtain the fingertip detection model.

[0107] Specifically, when training the improved yolov5 network using the training set, the network training parameters are set as follows: yolov5s.pt is used as the initial network weight, the training size is set to 640*640, the training iteration number is set to 50, the batch size is set to 16, and the initial learning rate is set to 0.01.

[0108] Since the yolov5 network will grid the input picture, divide it into N*N grids, and then predict the rectangular frame, confidence and classification probability of each grid, when training the model, the calculation of the loss function is mainly divided into three parts of loss.

[0109] Specifically, the loss function calculation includes rectangular frame loss calculation, confidence loss calculation and classification loss calculation.

[0110] The rectangular frame loss calculation calculates the rectangular frame loss by using the CIoU loss function, and the calculation formula is as follows:

[0111]

[0112]

[0113]

[0114]

[0115]

[0116] wherein S1 is the intersection area of the real frame and the predicted frame, S2 is the union area of the real frame and the predicted frame, ω t is the width of the real frame, h t is the height of the real frame, ω p is the width of the predicted frame, h p is the height of the predicted frame, a is the center point of the real frame, b is the center point of the predicted frame, δ 2 (a, b) is the Euclidean distance between a point and b point, λ is a weight coefficient, μ is the correlation coefficient of the length-width ratio of the real frame and the length-width ratio of the predicted frame, L CIoU is the rectangular frame loss of a single predicted frame.

[0117] The confidence loss calculation calculates the confidence loss by using the binary cross-entropy loss function, and the calculation formula is as follows:

[0118] L conf = -vlnm-(1-v)ln(1-m);

[0119] wherein v is the confidence label, m is the predicted confidence, L conf is the confidence loss of a single predicted frame.

[0120] The classification loss calculation calculates the classification loss using a binary cross-entropy loss function, and the calculation formula is as follows:

[0121] L class = -εlnp-(1-ε)ln(1-p);

[0122] Wherein, ε is the label probability, p is the prediction probability, L class is the classification loss of a single prediction box.

[0123] The calculation formula of the total loss L of the loss function is as follows:

[0124]

[0125] Wherein, ω1, ω2 and ω3 are the weights of the rectangular frame loss, the classification loss and the confidence loss in the total loss L, n is the number of positive samples, m is the number of negative samples, and e is the weight of the confidence loss of the positive sample in all confidence losses.

[0126] The training set is input into the improved yolov5 network for training, the prediction value is output through forward propagation, the loss function calculates the difference between the prediction value and the true value, the difference between the prediction value and the true value is the loss value, when the loss value stops decreasing, the training is stopped, and the fingertip detection model is output.

[0127] In some embodiments, after obtaining the fingertip detection model, the fingertip detection model needs to be evaluated and tested, including the following steps:

[0128] The fingertip detection model is evaluated using model evaluation indicators, including precision P, recall R and average precision mean mAP, and the formula is as follows:

[0129]

[0130]

[0131]

[0132]

[0133] Wherein, T is the number of correct predictions as positive samples, F is the number of false predictions as positive samples, N is the number of false predictions as negative samples, P(R) is the relationship function of precision P and recall R, AP is the average precision, AP k k represents the average precision of the kth target, and n represents the number of target categories.

[0134] The test set is input into the fingertip detection model for testing.

[0135] In order to test the effectiveness of the fingertip detection model based on the improved yolov5 network, an ablation experiment is performed on the test set and the fingertip detection model, and Table 1 shows the results of the ablation experiment:

[0136] Original yolov5 model BiFPN replaces PANet model C3EPSA replaces C3 model Improved yolov5 model P 95.6％ 95.3％ 95.1％ 95％ R 88.4％ 89％ 88.5％ 89.1％ mAP 90.7％ 90.7％ 91.3％ 91.5％

[0137] As shown in the above table, the fingertip detection model based on the improved yolov5 network has an accuracy of more than 95% in fingertip detection, which is 0.8% higher than the original in average precision.

[0138] Step S50: Collecting the trajectory of the fingertip movement based on the monocular camera, obtaining the real-time two-dimensional video image of the air handwriting text, and sequentially inputting each frame of the real-time two-dimensional video image of the air handwriting text into the fingertip detection model to obtain a plurality of target boxes containing fingertips.

[0139] Step S60: Based on the information of the plurality of target boxes containing fingertips and the coordinate system virtual sliding technology, obtaining a fingertip coordinate sequence composed of fingertip coordinates of each frame of the real-time two-dimensional video image of the air handwriting text, connecting each fingertip coordinate in the fingertip coordinate sequence to form the air handwriting text.

[0140] In some embodiments, the fingertip coordinate sequence composed of fingertip coordinates of each frame of the real-time two-dimensional video image of the air handwriting text is obtained based on the information of the plurality of target boxes containing fingertips and the coordinate system virtual sliding technology, and each fingertip coordinate in the fingertip coordinate sequence is connected to form the air handwriting text, including the following steps:

[0141] Step S601: Based on the left upper vertex position coordinates of each of the plurality of target boxes containing fingertips, the width of the target box, and the height of the target box, coordinate conversion is performed to obtain the fingertip coordinates of the initial real-time two-dimensional video image of each frame of the air handwriting text.

[0142] The coordinate conversion can be represented by the following formula:

[0143]

[0144] Wherein, the left upper vertex position coordinates of the target box of a certain frame of the real-time two-dimensional video image of the air handwriting text are (x lt , y lt ), the width of the target box is w, and the height of the target box is h, then the fingertip coordinates of a certain frame of the real-time two-dimensional video image of the air handwriting text are (x, y), and the conversion effect is as shown in Figure 8 .

[0145] Step S602: Based on the fingertip coordinates of the real-time two-dimensional video image of the initial frame of the handwritten text in the air and the coordinate system virtual sliding technique, obtain the fingertip coordinate sequence of the real-time two-dimensional video image of the handwritten text in the air. The coordinate system virtual sliding technique is as follows:

[0146] When obtaining the fingertip coordinates of the real-time 2D video image of the handwritten text in the air in the current frame, the horizontal coordinate of the fingertip coordinates of the real-time 2D video image of the handwritten text in the air in each frame before the current frame needs to be shifted to the left while keeping the vertical coordinate unchanged.

[0147] Wherein, when the fingertip coordinates (x) of frame t are detected t y t When ), for the preceding (t-1) fingertip coordinates, the fingertip coordinate sequence Q1=[x1,x2,...,x t-1 ;y1,y2,...,y t-1 Update the fingertip coordinate sequence Q1 = [x1, x2, ..., x...] t-1 ;y1,y2,...,y t-1 The x-coordinates of the (t-1) fingertip coordinates within the [element] are shifted to the left while the y-coordinates remain unchanged, which can be expressed by the following formula:

[0148] (x′ i y′ i )=(x i +v, y i )

[0149] Among them, (x i y i Let represent the coordinate of the i-th fingertip in the fingertip coordinate sequence Q1, where i = 1, 2, ..., t-1. (x' i y' i ) represents the coordinate of the i-th fingertip after the sliding process, and v represents the sliding speed.

[0150] After the update, the new sequence of the first (t-1) fingertip coordinates is obtained:

[0151] Q′1=[x′1,x′2,...,x′ t-1 ;y′1,y′2,...,y′ t-1 ];

[0152] Using the updated sequence of the first (t-1) fingertip coordinates Q'1 and the fingertip coordinates (x) of the t-th frame t y t To obtain the fingertip coordinate sequence of a real-time two-dimensional video image of handwritten text in the air:

[0153] Q = [x′1, x′2, ..., x′] t-1x t ; y'1, y'2,..., y' t-1 , y t ].

[0154] Step S603: connecting each fingertip coordinate in the fingertip coordinate sequence Q of the real-time two-dimensional video image of the air handwriting text in turn to form the air handwriting text.

[0155] In this embodiment, as shown in the figure, each fingertip coordinate in the fingertip coordinate sequence Q of the real-time two-dimensional video image of the air handwriting text is connected in turn with a solid line to form a track, and the track is the air handwriting text. Figure 9

[0156] Although the present application has been disclosed in the above preferred embodiments, it is not intended to limit the present application. Those skilled in the art can make various modifications and improvements without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be subject to the scope defined by the claims.​

Claims

1. A method for acquiring handwritten text in the air based on an improved YOLOv5 network and monocular vision, characterized in that, The method includes the following steps: Obtain several fingertip images containing fingertips, and construct an initial fingertip dataset based on these images; The initial fingertip dataset is preprocessed to obtain the target fingertip dataset, and the target fingertip dataset is divided into training set and test set according to the proportion. The YOLOv5 network is improved based on the efficient pyramid segmentation attention module EPSA and the weighted bidirectional feature pyramid network BiFPN. The improvements are as follows: the bottleneck module in the C3 module is replaced with the efficient pyramid segmentation attention module EPSA to obtain the C3EPSA module; the C3 module connected to the SPPF module in the backbone network of the YOLOv5 network is replaced with the C3EPSA module; the C3 module in the deep output layer of the neck network of the YOLOv5 network is replaced with the C3EPSA module to enhance the feature extraction capability of the YOLOv5 network; and the Concat module in the neck network of the YOLOv5 network is replaced with the weighted bidirectional feature pyramid network BiFPN to better integrate the feature maps of different levels obtained from the backbone network. The training set is fed into the improved YOLOv5 network for training and the predicted value is output. The loss value is calculated using the loss function. When the loss value stops decreasing, the fingertip detection model is obtained. Based on the acquisition of the trajectory of fingertip movement by a monocular camera, real-time two-dimensional video images of handwritten text in the air are obtained. Each frame of the real-time two-dimensional video image of handwritten text in the air is then input into the fingertip detection model to obtain several target boxes containing the fingertip. Based on information from several target boxes containing fingertips and virtual coordinate system sliding technology, a fingertip coordinate sequence is obtained from the real-time two-dimensional video image of each frame of aerial handwritten text. Each fingertip coordinate in the fingertip coordinate sequence is then connected to form the aerial handwritten text.

2. The method for acquiring handwritten text in the air based on an improved YOLOv5 network and monocular vision as described in claim 1, characterized in that, Some of the fingertip images were obtained through the internet, while the other part of the fingertip images were obtained by capturing images of different people's fingers under different backgrounds, lighting conditions, angles, and distances using a monocular camera.

3. The method for acquiring handwritten text in the air based on an improved YOLOv5 network and monocular vision as described in claim 1, characterized in that, The process of preprocessing the initial fingertip dataset to obtain the target fingertip dataset, and then dividing the target fingertip dataset into training and testing sets proportionally, includes the following steps: The object detection dataset annotation tool lableimg was used to annotate several fingertip images in the initial fingertip dataset, and the positions of the fingertips in the images were marked with rectangles; The initial fingertip dataset after annotation is expanded using mosaic stitching, affine transformation, and segmentation and padding data augmentation methods to obtain the target fingertip dataset. The target fingertip dataset was divided into training and testing sets in an 8:2 ratio.

4. The method for acquiring handwritten text in the air based on an improved YOLOv5 network and monocular vision as described in claim 1, characterized in that, The specific operation method of the High-Efficiency Pyramid Segmentation Attention Module (EPSA) is as follows: The first input feature map of the efficient pyramid segmentation attention module EPSA After passing through a Conv1×1 module, the second input feature map is obtained. ; For the second input feature map According to the second input feature map The number of channels will affect the second input feature map. Average division into Each segmentation feature map ; against Each segmentation feature map ,use Grouped convolution with different kernels extracts the first input feature map. Feature maps at different scales ,in, ; Feature maps of different scales The data is input into the SEWeight module for learning, extracting feature maps at different scales. Channel attention is used to obtain feature maps at different scales. Channel attention vector ,in, ; use The function applies to feature maps at each different scale. Channel attention vector Perform weight recalibration to obtain the weight vector. ,in, ; Feature maps of corresponding scales With weight vector Perform Hadamard product operations to obtain feature maps at different scales after channel attention weighting. , take S feature maps Perform concatenation and fusion to obtain multi-scale channel attention-weighted feature maps. ,in, ; Feature maps after multi-scale channel attention weighting After inputting the Conv1×1 module, it is compared with the first input feature map. Perform a summation operation to obtain an output feature map that contains multi-scale spatial information and cross-channel attention. .

5. The method for acquiring handwritten text in mid-air based on an improved YOLOv5 network and monocular vision as described in claim 1, characterized in that, When training the improved YOLOv5 network using the training set, the network training parameters are set as follows: Using yolov5s.pt as the initial network weights, the training size was set to 640*640, the number of training iterations was set to 50, the batch size was set to 16, and the initial learning rate was 0.

01.

6. The method for acquiring handwritten text in mid-air based on an improved YOLOv5 network and monocular vision as described in claim 1, characterized in that, The loss function calculation includes bounding box loss calculation, confidence loss calculation, and classification loss calculation; The rectangular box loss is calculated using the CIoU loss function, and the calculation formula is as follows: ； ； ； ； ； in, This represents the intersection area of ​​the ground truth bounding box and the predicted bounding box. Let be the area of ​​the union of the ground truth bounding box and the predicted bounding box. The width of the actual bounding box. The height of the actual bounding box. The width of the prediction box, Let be the height of the predicted bounding box, 'a' be the center point of the ground truth bounding box, and 'b' be the center point of the predicted bounding box. Let be the Euclidean distance between points a and b. These are the weighting coefficients. This is the correlation coefficient between the aspect ratios of the ground truth bounding box and the aspect ratios of the predicted bounding box. The rectangular box loss is for a single predicted box; The confidence loss is calculated using the binary cross-entropy loss function, and the formula is as follows: ； in, v For confidence level labels, m To predict confidence levels, The confidence loss for a single predicted bounding box; The classification loss is calculated using the binary cross-entropy loss function, and the formula is as follows: ； in, For label probability, p To predict probabilities, The classification loss is for a single predicted bounding box; The formula for calculating the total loss L of the loss function is as follows: ； in, , as well as The bounding box loss, classification loss, and confidence loss are respectively included in the total loss. The weight of the middle, n The number of samples that are classified as positive. m The number of samples classified as negative. e The weight of the confidence loss for positive samples relative to the total confidence loss.

7. The method for acquiring handwritten text in mid-air based on an improved YOLOv5 network and monocular vision as described in claim 1, characterized in that, After obtaining the fingertip detection model, it needs to be evaluated and tested, including the following steps: The fingertip detection model was evaluated using model evaluation metrics, including accuracy. P Recall rate R and mean precision mAP The formula is as follows: ； ； ； ； Where T is the number of correctly predicted positive samples, F is the number of incorrectly predicted positive samples, N is the number of incorrectly predicted negative samples, P(R) is the function relating precision P and recall R, and AP is the mean precision. Indicates the first k The average precision of target classes, where n represents the number of target classes; The test set is input into the fingertip detection model for testing.

8. The method for acquiring handwritten text in the air based on an improved YOLOv5 network and monocular vision as described in claim 1, characterized in that, The method of obtaining a fingertip coordinate sequence composed of fingertip coordinates from a real-time two-dimensional video image of each frame of aerial handwritten text based on information from several target boxes containing fingertips and virtual coordinate system sliding technology, and connecting each fingertip coordinate in the fingertip coordinate sequence to form aerial handwritten text, includes the following steps: Based on the coordinates of the top left vertex of each of several target boxes containing fingertips, the width of the target box, and the height of the target box, coordinate transformation is performed to obtain the fingertip coordinates of the real-time two-dimensional video image of the initial frame of the handwritten text in the air. Based on the fingertip coordinates of the real-time two-dimensional video image of the initial frame of the handwritten text in the air, and using a virtual coordinate system sliding technique, a sequence of fingertip coordinates from the real-time two-dimensional video image of the handwritten text in the air is obtained. The virtual coordinate system sliding technique is as follows: When obtaining the fingertip coordinates of the real-time 2D video image of the handwritten text in the air in the current frame, the horizontal coordinate of the fingertip coordinates of the real-time 2D video image of the handwritten text in the air in each frame before the current frame needs to be shifted to the left while keeping the vertical coordinate unchanged. By sequentially connecting the fingertip coordinates of the real-time two-dimensional video image of the handwritten text in the air, the handwritten text in the air is formed.

Images