Pen holding posture detection method based on multi-view attention fusion

Abstract

The embodiment of the invention discloses a pen holding posture detection method based on multi-view attention fusion, and relates to the technical field of computer vision, and the method comprises the steps: collecting first image data of a to-be-detected target through an infrared image collection device, and collecting second image data through a 3D image collection device; a first hand area image is extracted based on the first image data, a second hand area image is extracted based on the second image data, the first hand area image and the second hand area image are input into a trained pen holding posture recognition model, a pen holding posture feature vector is extracted, and a pen holding posture detection result is determined based on the pen holding posture feature vector. According to the method provided by the invention, the kinematics constraint is added in the extracted pen holding posture feature vector, and the joint angle and the joint node are introduced through the transform module, so that the feature vector output by the detection model can reflect a more real and reasonable hand posture in a pen holding state, and the accuracy of the holding posture estimated by the pen holding posture recognition model is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method for detecting pen-holding posture based on multi-view attention fusion. Background Art

[0002] Correct pen-holding posture is crucial to cultivating personal writing habits, improving writing skills, and preventing hand fatigue. Incorrect pen-holding posture may cause problems such as hand fatigue, muscle pain, and poor hand coordination. If the wrong pen-holding posture is used for a long time, it may have an adverse effect on the growth and development of the hands, which in turn affects their writing ability and daily life ability. Traditional posture guidance relies on manual supervision, which is not only inefficient but also unable to provide real-time feedback.

[0003] Related technologies can use mainstream deep learning convolutional neural network models such as VGG, ResNet and DenseNet to analyze images and indicate whether the pen holding posture in a given image is wrong. However, this method has the following challenges in detecting students' pen holding posture:

[0004] 1. The hand appearance of different learners varies greatly, and some images contain a certain degree of hand occlusion and noise; 2. The similarity between hand parts; 3. The positional relationship between the hand and the writing pen cannot be accurately distinguished.

[0005] Therefore, in daily learning life, accurately detecting and identifying the pen-holding posture is a problem that needs to be solved urgently. Summary of the invention

[0006] The embodiment of the present application provides a method for detecting pen-holding posture based on multi-view attention fusion, so as to solve the defects of the above-mentioned related technologies. The technical solution is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for detecting a pen-holding posture based on multi-view attention fusion, comprising:

[0008] The infrared image acquisition device is used to acquire first image data of at least one target to be detected, and the 3D image acquisition device is used to acquire second image data of the at least one target to be detected; wherein the infrared image acquisition device and the 3D image acquisition device are arranged at different positions, and the image frames acquired by the infrared image acquisition device and the 3D image acquisition device at the same time have an overlapping area;

[0009] performing geometric transformation and data enhancement processing on each image frame in the first image data and each image frame in the second image data in sequence, extracting a first hand region image based on each image frame in the first image data after the data enhancement processing, and extracting a second hand region image based on each image frame in the second image data after the data enhancement processing;

[0010] The first hand area image and the second hand area image are input into a trained pen-holding posture recognition model, so that the trained pen-holding posture recognition model extracts the pen-holding posture feature vector of the target to be detected based on the first hand area image and the second hand area image, and determines the pen-holding posture detection result of the target to be detected based on the pen-holding posture feature vector.

[0011] In an optional solution of the first aspect, determining the pen-holding posture detection result of the target to be detected based on the pen-holding posture feature vector further includes:

[0012] Obtaining the pen-holding posture recognition result of the target to be detected at each moment in a preset detection period;

[0013] Compare the pen-holding posture detection result at each moment with the preset standard pen-holding posture respectively, and calculate the Hausdorff distance between each feature point in the image frame corresponding to the pen-holding posture detection result and each feature point in the preset standard pen-holding posture;

[0014] The pen-holding posture detection result of the target to be detected at each moment in a preset detection period is output according to the comparison result of the Hausdorff distance of all feature points with the Hausdorff distance threshold.

[0015] In an optional solution of the first aspect, after outputting the pen holding posture detection result of the target to be detected at each moment in a preset detection period according to the comparison result of the Hausdorff distance of all feature points with the Hausdorff distance threshold, it also includes:

[0016] Determine the proportion of the time period in which the Hausdorff distance of all the feature points is greater than the Hausdorff distance threshold in the detection time period, and determine and output the pen-holding posture detection result of the target to be detected within the preset detection time period based on the value of the proportion.

[0017] In an optional solution of the first aspect, after outputting the pen-holding posture detection result of the target to be detected within a preset detection period according to the numerical value of the ratio, the method further includes:

[0018] Determine the proportion range within which the proportion of the time period in which the Hausdorff distance of all the feature points is greater than the Hausdorff distance threshold falls in the detection time period, and output the corresponding pen holding posture adjustment scheme to the device end of the target to be detected based on the mapping relationship between the proportion range and the pen holding posture adjustment scheme.

[0019] In a second aspect, the embodiment of the present application further provides a pen-holding posture detection device based on multi-view attention fusion, comprising:

[0020] An image acquisition module, configured to acquire first image data of at least one target to be detected by an infrared image acquisition device, and to acquire second image data of the at least one target to be detected by a 3D image acquisition device; wherein the infrared image acquisition device and the 3D image acquisition device are arranged at different positions, and image frames acquired by the infrared image acquisition device and the 3D image acquisition device at the same time have an overlapping area;

[0021] a hand region extraction module, configured to perform geometric transformation and data enhancement processing on each image frame in the first image data and each image frame in the second image data in sequence, extract a first hand region image based on each image frame in the first image data after the data enhancement processing, and extract a second hand region image based on each image frame in the second image data after the data enhancement processing;

[0022] A pen-holding posture detection module is used to input the first hand area image and the second hand area image into a trained pen-holding posture recognition model, so that the trained pen-holding posture recognition model extracts the pen-holding posture feature vector of the target to be detected based on the first hand area image and the second hand area image, and determines the pen-holding posture detection result of the target to be detected based on the pen-holding posture feature vector.

[0023] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method provided in the first aspect of the embodiment of the present application or any one of the implementation methods of the first aspect is implemented.

[0024] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of the embodiment of the present application or any one of the implementations of the first aspect.

[0025] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0026] A pen-holding posture detection method based on multi-view attention fusion provided in an embodiment of the present application solves the problem of occlusion of hand image data by collecting data through dual cameras, and uses a denoising diffusion model to learn the hand distribution from images under the feature conditions of MobileNetV2. This enables the present application to determine the 3D structure of the student's hand when holding a pen from image features. The embodiment of the present application also adds kinematic constraints to the extraction of the pen-holding posture feature vector, and introduces restrictions on joint angles and joint nodes through the transformer module, so that the feature vector output by the detection model can reflect a more realistic and reasonable hand posture in the pen-holding state, thereby improving the accuracy of the 3D holding posture estimated by the pen-holding posture recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flow chart of a method for detecting pen-holding posture based on multi-view attention fusion according to an embodiment of the present application;

[0029] Figure 2 It is a flow chart of a method for detecting pen-holding posture based on multi-view attention fusion according to an embodiment of the present application;

[0030] Figure 3 It is a structural schematic diagram of a pen-holding posture detection device based on multi-view attention fusion provided in an embodiment of the present application;

[0031] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.

[0034] It should be noted that the terms "first\second" involved in the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those described or illustrated herein.

[0035] The present application is described in detail below with reference to specific embodiments.

[0036] Next, combine Figure 1 , introduces the method for detecting pen holding posture based on multi-view attention fusion provided by the embodiment of the present application. For details, please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart showing a method for detecting a pen-holding posture based on multi-view attention fusion provided by an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0037] S101, collecting first image data of at least one target to be detected by an infrared image acquisition device, and collecting second image data of the at least one target to be detected by a 3D image acquisition device;

[0038] S102, performing geometric transformation and data enhancement processing on each image frame in the first image data and each image frame in the second image data in sequence, extracting a first hand region image based on each image frame in the first image data after the data enhancement processing, and extracting a second hand region image based on each image frame in the second image data after the data enhancement processing;

[0039] S103, inputting the first hand area image and the second hand area image into a trained pen-holding posture recognition model, so that the trained pen-holding posture recognition model extracts the pen-holding posture feature vector of the target to be detected according to the first hand area image and the second hand area image, and determines the pen-holding posture detection result of the target to be detected based on the pen-holding posture feature vector.

[0040] In some embodiments, in S101, image information can be obtained by respectively setting an infrared image acquisition device I_camera and a 3D image acquisition device DVS_camera at both ends of the desktop, the infrared image acquisition device acquires first image data Img corresponding to the target to be detected, and the 3D image acquisition device acquires second image data DImg corresponding to the target to be detected.

[0041] It can be understood that the infrared image acquisition device and the 3D image acquisition device are arranged at different positions, and the image frames acquired by the infrared image acquisition device and the 3D image acquisition device at the same time have overlapping areas.

[0042] In some embodiments, geometric transformation and data enhancement processing may be performed sequentially on each image frame in the first image data and each image frame in the second image data, including:

[0043] Preprocessing steps including in-plane rotation ([-180, 180] degrees), 3D scaling ([0.9, 1, 1]) and 3D translation ([-8, 8] mm) are performed on each image frame in the first image data and each image frame in the second image data, and data enhancement is performed on the two image data respectively. For example, contrast loss (SimCLR) can be used for data enhancement for each image frame in the first image data, and NewDropout function can be used for data enhancement for each image frame in the second image data.

[0044] In some embodiments, the step of performing data enhancement based on contrastive loss (SimCLR) includes:

[0045] The contrast loss function (SimCLR) is to obtain two different data enhancement results for each image ( and ), using the MobileNetV2 network model to extract features from the input image data (z i and z j ) and superimpose a hidden fully connected layer to reduce the feature dimension, further extract the low-level features and high-level features of the hand and combine them to output high-resolution feature quantities; the high-level features can be understood as hand features, and the low-level features can be understood as background features.

[0046] Specifically, the contrast loss function can use cosine similarity to measure the similarity s between two data-enhanced images: i,j , the formula is as follows:

[0047]

[0048] Using the softmax function to obtain the probability loss function of the two data-enhanced images being similar is:

[0049]

[0050] Take the average of the sum of all image losses:

[0051]

[0052] In some embodiments, the step of performing data augmentation using the NewDropout function includes:

[0053] The data enhancement function NewDropout is defined as the function Tα:

[0054] R W×H →R W×H ;

[0055] Among them, the parameter α∈[0, 1] is used to control the strength of data enhancement.

[0056] First, a uniform sampling can be performed from each image frame in the second image data within the range of [0, α] with a probability γ, and further, a group of pixels Q can be obtained by uniform sampling.

[0057] By deleting selected pixels, an enhanced depth image is obtained. As shown below:

[0058]

[0059] Here, p represents any pixel in the image and C represents a constant value assigned to pixels outside the hand region.

[0060] Further, in S102, based on each frame of the preprocessed first image data and each frame of the second image data, hand target detection is performed respectively to determine a hand region image, specifically including:

[0061] Specifically, the position of the hand can be transformed from the coordinate position in the real space to the pixel coordinate position in the camera frame. For example, a perspective camera model with a projection matrix P = K[R|t] can be selected, where Is an intrinsic matrix, R represents the rotation matrix of the camera, and t represents the translation of the camera, thereby transforming the position of the hand from the coordinate position in the real space to the pixel coordinate position in the camera frame.

[0062] It can be understood that a 3D hand model with N joints can be represented as a matrix C of size 3×N, and each joint point is stacked in a column of the matrix C. Thus, the corresponding two-dimensional projection position of the hand in the image frame can be extracted.

[0063] Furthermore, each frame image in the first image data is cropped to obtain the first hand area image, and each frame image in the second image data is cropped to obtain the second hand area image. The corresponding area images can also be cropped to a fixed size of a preset size, such as 128×128.

[0064] In some embodiments, Figure 2 As shown, in S103, the first hand area image and the second hand area image can be input into a trained pen-holding posture recognition model, so that the trained pen-holding posture recognition model extracts the pen-holding posture feature vector of the target to be detected according to the first hand area image and the second hand area image. The pen-holding posture feature vector can be extracted through two stages, specifically including:

[0065] In the first stage, the feature vector F can be obtained by the RGB image processing SubNet IRC branch and the 3D point cloud image processing ECNet branch. Infrared and F Event ;

[0066] The second stage can be used to transform the feature vector F Infrared and F Event Data fusion is performed to obtain a pen-holding posture feature vector, and the pen-holding posture detection result of the target to be detected is further determined based on the pen-holding posture feature vector through a MANO model.

[0067] Specifically, the first stage includes:

[0068] For the SubNetIRC branch, the MobileNetV2 network is used as the encoder, and the encoder is defined as the RGB image in the first image data input to the output feature quantity f:R W×H →R c×h×w The nonlinear mapping of h, w, and c represents the height, width, and number of channels of the output feature quantity, respectively. The MobileNetV2 network model is used to extract features from the input image data and a hidden fully connected layer is superimposed to reduce the feature dimension.

[0069] The input image data is further processed by MobileNetV2 to extract and combine the low-level and high-level features of the hand, output high-resolution feature quantities, and pass the feature quantities to the diffusion model.

[0070] Assume that the initial state of the input diffusion model is x0. The diffusion model adopts the U-Net architecture. The encoder captures the context information of the image and gradually injects noise into it to obtain x t, reduce the spatial dimension and increase the number of channels until the data becomes pure noise x T ,The decoder reconstructs the image from the high-level features passed by the encoder, gradually restoring the original state of the data, increasing the spatial dimension and reducing the number of channels. In each step of the decoder, the feature map from the corresponding encoder layer is connected to the upsampled feature map through a skip connection.

[0071] For training diffusion models, the goal is usually to minimize the Kullback-Leibler divergence. First, set where σ 2 ≈β t Additionally, using the distribution:

[0072]

[0073] Train the inverse process average function approximator μ θ To predict The training of the denoising model is done by taking gradient descent steps on the difference between the sampled noise and the predicted noise, and the training of the denoising model is done by taking gradient descent steps on the difference between the sampled noise and the predicted noise, and the formula is as follows:

[0074]

[0075] in is the random sampling noise, t is uniformly randomly sampled in the range of 0 to T, ∈ θ The U-Net model conditioned on time step t and image features f is used to predict Gaussian noise, β t is a parameter that controls the amount of noise.

[0076] During inference, pure Gaussian noise is concatenated with image features, and given a denoising time step, the U-Net model gradually denoises between two consecutive time steps. Therefore, the recursive equation for generating a 3D skeleton from an image is:

[0077]

[0078] Among them from Start calculation If t>0, then Otherwise z=0.

[0079] Then the denoised feature vector is obtained.

[0080] Furthermore, the denoised feature vector output by the diffusion model is input into the UV module, the attention enhancement module, and the transformer module respectively.

[0081] Among them, the UV module can be used to calculate the attention feature map Att of each hand joint in the denoised image. uv ∈R i×h×w ,use Represents the attention feature map corresponding to the i-th joint.

[0082] To ensure that each attention feature map focuses on the pixels where the corresponding joint appears, firstly, the attention feature map Att uv Normalize the spatial softmax layer to get the corresponding heat map as follows:

[0083]

[0084] Among them, σ is the spatial softmax layer, heat map represents the probability of the i-th joint appearing at each pixel position, and Ω represents the attention feature map spatial domain.

[0085] Specifically, the two-dimensional position of the i-th hand joint in the image can be calculated by integration operation as follows:

[0086]

[0087] in represents the identification coordinates of the i-th joint in the deep image space. Supervision is applied to all attention feature maps To minimize the average distance L1, apply the formula:

[0088]

[0089] Among them (U i , V i ) represents the actual coordinates of the i-th hand joint in the image in the RGB image space.

[0090] In some embodiments, the attention enhancement module aims to calculate a more flexible attention feature map to enhance Att uv Computed attention feature maps to facilitate recognition of joint depth values.

[0091] Specifically, the output feature of the encoder is used as input to calculate the attention feature map Att of each joint mix ∈R i×h×w , Represents the attention feature map corresponding to the i-th joint. uv In contrast, the attention map has no external constraints (supervision) and is used to identify the depth value of each joint in different scenes.

[0092] Furthermore, two attention feature maps and are complementary, The network attention is shifted to the pixel where the i-th joint occurs, and Help the network to notice non-joint pixels, which may contain useful information for identifying the depth value of the i-th joint. Understandably, the attention feature map Attention feature map for characterizing the pixels at the hand joint positions The area outside the pixels representing the hand joint positions is used to gain attention.

[0093] The two attention feature maps are fused as follows:

[0094]

[0095] Among them, β i ∈[0, 1], β i is a parameter of the model, used to control the effect of each attention feature map on the fusion attention feature map The weight of the fused attention feature map can be used to guide the feature pooling of subsequent feature value recognition.

[0096] In some embodiments, the transformer module, also known as the temporal attention module, can extract the hand joint information when a continuous image input sequence is available, so as to utilize the temporal information on the continuous frame sequence, perform feature pooling on the above information and add a forward kinematics constraint layer to obtain the final feature vector F Infrared .

[0097] Specifically, the pen holding posture can be accurately predicted through a forward kinematics layer. The forward kinematics layer is a non-parametric layer of the network, which is implemented by traversing the tree graph through the breadth-first search algorithm (BFS). Each hand node i has an Euler angle relative to another node and at and at the limit Internal, translation offset The relative rotation of the nodes is obtained by converting the Euler angles into a rotation matrix:

[0098] R' i =φ(e i );

[0099] Application Mapping The relative rotation matrix R' can have 0 to 3 degrees of freedom depending on the parameterization of the node. The rotation of the hand node in the camera frame is denoted by R i , the position in three-dimensional space is denoted as p i . Therefore the root has a relative rotation and offset with respect to the camera frame, namely:

[0100] R0=R'0;

[0101]

[0102] At each stage of the BFS tree traversal, the positions and orientations of all other nodes in space are found by a recursive rule as follows:

[0103] R i =R j R' i ;

[0104] p i =R i o i +p j ;

[0105] Among them, the edge (i, j) represents the edge from the parent node j to the child node i. The position of each joint depends on the parent node, while the root node is unconstrained and has three degrees of freedom. The extracted feature vector can be expressed as:

[0106]

[0107] In this way, the rationality of the extracted hand features can be ensured to be consistent with the physiological characteristics of the human body.

[0108] In some embodiments, the feature vector F is obtained by processing the ECNet branch through the 3D point cloud image Event , including:

[0109] Based on the input second hand region image, key feature points that may correspond to the two-dimensional joint positions are first extracted, and a single-channel heat map H is predicted.

[0110] Ground truth heatmap H * is obtained by applying a 2D Gaussian variance kernel σ to each ground-truth 2D joint location.

[0111] Further, an appearance and spatial encoding can be calculated for each detected key feature point to represent the key point as input to the next stage. Image features are extracted from the decoder of the U-Net network, and bilinear interpolation is used to sample the multi-layer feature maps of the U-Net decoder at the standardized key feature point locations and concatenate them to form a feature vector, which is then reduced to a 224-D encoding vector using a 3-layer MLP.

[0112] For the spatial coding, a 32-D sinusoidal position coding similar to can be obtained, corresponding to the two-dimensional position of the key feature point.Finally, the present invention connects the 224-D coding vector and the 32-D sinusoidal position coding to form a 256-D vector representation of the key feature point.

[0113] Specifically, for each key feature point Ki there is a corresponding encoding vector Fi∈R256. These vectors are used as the input of the multi-layer, multi-head self-attention module with the NSA layer. The identity of the key feature point k is defined as (h k ,j k ), where h k is the hand identity (left or right), j k It is a joint index.

[0114] In some embodiments, for the key feature points that are detected incorrectly, the present invention can also use an additional "background" identifier to predict the type of the key feature points using a feedforward network (FFN) consisting of a 2-layer MLP, a linear projection layer, and a softmax layer, and the cross entropy loss is as follows:

[0115]

[0116] in, is the true value relative to the horizontal plane, and CE represents the cross entropy loss. Through such steps, the type of each pixel in the image can be determined to determine whether it is a hand joint pixel or other background nodes other than the hand joint pixel.

[0117] Specifically, in the second stage, the feature vector F Infrared and F Event To perform fusion, apply the formula:

[0118] F = σ(αF Infrared +βF Event );

[0119] Among them, α and β are the eigenvectors F Infrared and the eigenvector F Event The weight of .

[0120] Specifically, in S103, the fused feature vector F can be input into the MANO model training to obtain a visualized three-dimensional grip grid Hand Estimation , use Hausdorff distance to calculate its difference from the standard grip Hand GT The similarity.

[0121] The Hausdorff distance principle is as follows: d H (A,B)=max{sup(a∈A)inf(b∈B)d(a,b),sup(b∈B)(inf(a∈A)d(a,b)};

[0122] Where d(a, b) is the Euclidean distance between point a and point b. In short, the Hausdorff distance is the larger distance between the maximum value of the distance between a point in point set A and the nearest point in point set B and the maximum value of the distance between a point in point set B and the nearest point in point set A.

[0123] Specifically, the pen holding posture can be scored based on the pen holding posture detection result calculated by Hausdorff distance. The system learns from 10,000 samples, and then scores the obtained Hausdorff distance from 0 to 100 according to the normal distribution. That is, the Hausdorff distance is arranged in order from low to high, and the corresponding score is assigned in proportion, where the larger the Hausdorff distance, the lower the score, and the smaller the Hausdorff distance, the higher the score. Prompts are given for those with scores below a certain threshold (extremely incorrect pen holding posture), and student users are reminded to watch relevant videos to learn pen holding posture. For the excellent scoring pen holding posture, the alarm clock page displays the score in real time, and students can dynamically view the score curve and adjust the grip in real time.

[0124] In some embodiments, the pen-holding posture may be detected once at preset time intervals, for example, the pen-holding posture detection result at each time in the corresponding detection period may be determined every ten minutes.

[0125] In some embodiments, it is possible to determine the proportion range within which the proportion of the time period in which the Hausdorff distance of all feature points is greater than the Hausdorff distance threshold falls within the detection time period, and output the corresponding pen holding posture adjustment scheme to the device end of the target to be detected based on the mapping relationship between the proportion range and the pen holding posture adjustment scheme.

[0126] Specifically, the pen-holding posture adjustment plan can be understood as determining the joint node where the actual pen-holding posture deviates the most from the preset standard pen-holding posture based on the maximum value of the Hausdorff distance, and prompting learners to pay attention to the incorrect pen-holding posture problems at the corresponding nodes, so as to facilitate learners to grasp specific problems and help learners master a more standard pen-holding posture.

[0127] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0128] See next Figure 3 , is a schematic diagram of the structure of a pen-holding posture detection device based on multi-view attention fusion provided by an exemplary embodiment of the present application. The device can be implemented as all or part of a terminal through software, hardware, or a combination of both, and can also be integrated on a server as an independent module. The pen-holding posture detection device based on multi-view attention fusion in the embodiment of the present application can be applied to a terminal or a cloud. The device 30 includes an image acquisition module 301, a hand region extraction module 302, and a pen-holding posture detection module 303, wherein:

[0129] The image acquisition module 301 is used to acquire first image data of at least one target to be detected through an infrared image acquisition device, and to acquire second image data of the at least one target to be detected through a 3D image acquisition device; wherein the infrared image acquisition device and the 3D image acquisition device are arranged at different positions, and image frames acquired by the infrared image acquisition device and the 3D image acquisition device at the same time have an overlapping area;

[0130] The hand region extraction module 302 is used to perform geometric transformation and data enhancement processing on each image frame in the first image data and each image frame in the second image data in sequence, and obtain a first hand region image based on each image frame in the first image data after the data enhancement processing, and obtain a second hand region image based on each image frame in the second image data after the data enhancement processing;

[0131] The pen-holding posture detection module 303 is used to input the first hand area image and the second hand area image into a trained pen-holding posture recognition model, so that the trained pen-holding posture recognition model extracts the pen-holding posture feature vector of the target to be detected based on the first hand area image and the second hand area image, and determines the pen-holding posture detection result of the target to be detected based on the pen-holding posture feature vector.

[0132] It should be noted that when the device 30 provided in the above embodiment executes the method for detecting the pen-holding posture based on multi-view attention fusion, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the embodiment of the method for detecting the pen-holding posture based on multi-view attention fusion belong to the same concept. The implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0133] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the program.

[0134] See also Figure 4 , which is a structural block diagram of an electronic device provided in an embodiment of the present application.

[0135] like Figure 4 As shown, the electronic device 400 includes: a processor 401 and a memory 402 .

[0136] In the embodiment of the present application, the processor 401 is the control center of the computer system, which can be a processor of a physical machine or a processor of a virtual machine. The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array).

[0137] The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also called a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0138] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the method in the embodiment of the present application.

[0139] In some embodiments, the electronic device 400 further includes: a peripheral device interface 403 and at least one peripheral device 404. The processor 401, the memory 402 and the peripheral device interface 403 can be connected via a bus or a signal line. Each peripheral device 404 can be connected to the peripheral device interface 403 via a bus, a signal line or a circuit board. Specifically, the peripheral device 404 includes: a display screen, a camera and an audio circuit. The peripheral device interface 403 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 401 and the memory 402.

[0140] In some embodiments of the present application, the processor 401, the memory 402, and the peripheral device interface 403 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 401, the memory 402, and the peripheral device interface 403 can be implemented on a separate chip or circuit board. This embodiment of the present application does not specifically limit this.

[0141] The electronic device structure block diagram shown in the embodiment of the present application does not constitute a limitation on the electronic device 400. The electronic device 400 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0142] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method of any of the above embodiments are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting pen-holding posture based on multi-view attention fusion, characterized in that: include: The infrared image acquisition device is used to acquire first image data of at least one target to be detected, and the 3D image acquisition device is used to acquire second image data of the at least one target to be detected; wherein the infrared image acquisition device and the 3D image acquisition device are arranged at different positions, and the image frames acquired by the infrared image acquisition device and the 3D image acquisition device at the same time have an overlapping area; performing geometric transformation and data enhancement processing on each image frame in the first image data and each image frame in the second image data in sequence, extracting a first hand region image based on each image frame in the first image data after the data enhancement processing, and extracting a second hand region image based on each image frame in the second image data after the data enhancement processing; The first hand area image and the second hand area image are input into a trained pen-holding posture recognition model, so that the trained pen-holding posture recognition model extracts the pen-holding posture feature vector of the target to be detected based on the first hand area image and the second hand area image, and determines the pen-holding posture detection result of the target to be detected based on the pen-holding posture feature vector.

2. According to the method of claim 1, the method is characterized in that: The step of determining the pen holding posture detection result of the target to be detected based on the pen holding posture feature vector further includes: Obtaining the pen-holding posture recognition result of the target to be detected at each moment in a preset detection period; Compare the pen-holding posture detection result at each moment with the preset standard pen-holding posture respectively, and calculate the Hausdorff distance between each feature point in the image frame corresponding to the pen-holding posture detection result and each feature point in the preset standard pen-holding posture; The pen-holding posture detection result of the target to be detected at each moment in a preset detection period is output according to the comparison result of the Hausdorff distance of all feature points with the Hausdorff distance threshold.

3. According to claim 2, a method for detecting pen-holding posture based on multi-view attention fusion, characterized in that: After outputting the pen holding posture detection result of the target to be detected at each moment in the preset detection period according to the comparison result of the Hausdorff distance of all feature points with the Hausdorff distance threshold, the method further includes: Determine the proportion of the time period in which the Hausdorff distance of all the feature points is greater than the Hausdorff distance threshold in the detection time period, and determine and output the pen-holding posture detection result of the target to be detected within the preset detection time period based on the value of the proportion.

4. The method for detecting pen-holding posture based on multi-view attention fusion according to claim 3, characterized in that: After outputting the pen-holding posture detection result of the target to be detected within a preset detection period according to the value of the ratio, the method further includes: Determine the proportion range within which the proportion of the time period in which the Hausdorff distance of all the feature points is greater than the Hausdorff distance threshold falls in the detection time period, and output the corresponding pen holding posture adjustment scheme to the device end of the target to be detected based on the mapping relationship between the proportion range and the pen holding posture adjustment scheme.

5. A pen-holding posture detection device based on multi-view attention fusion, characterized in that: include: An image acquisition module, configured to acquire first image data of at least one target to be detected by an infrared image acquisition device, and to acquire second image data of the at least one target to be detected by a 3D image acquisition device; wherein the infrared image acquisition device and the 3D image acquisition device are arranged at different positions, and image frames acquired by the infrared image acquisition device and the 3D image acquisition device at the same time have an overlapping area; a hand region extraction module, configured to perform geometric transformation and data enhancement processing on each image frame in the first image data and each image frame in the second image data in sequence, extract a first hand region image based on each image frame in the first image data after the data enhancement processing, and extract a second hand region image based on each image frame in the second image data after the data enhancement processing; A pen-holding posture detection module is used to input the first hand area image and the second hand area image into a trained pen-holding posture recognition model, so that the trained pen-holding posture recognition model extracts the pen-holding posture feature vector of the target to be detected based on the first hand area image and the second hand area image, and determines the pen-holding posture detection result of the target to be detected based on the pen-holding posture feature vector.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images