Aerial handwriting man-machine interaction method and device based on SAM2 large model, electronic equipment and storage medium

Abstract

The invention relates to an air handwriting man-machine interaction method based on an SAM2 large model, which comprises the following steps: acquiring a two-dimensional video image sequence, importing the two-dimensional video image sequence into the SAM2 large model to obtain a hand region segmentation mask, extracting a hand contour based on the hand region segmentation mask, and separating palm and finger regions. Positioning in the finger area based on a multi-angle fingertip template matching algorithm and acquiring coordinates of a fingertip; and during writing, tracking and recording a coordinate sequence of fingertip movement so as to generate an aerial writing text track based on a coordinate system virtual sliding technology. The invention further relates to an air handwriting man-machine interaction device based on the SAM2 large model. The air handwriting man-machine interaction device comprises an acquisition module, an input module, a processing module and an output module. The invention further relates to electronic equipment and a storage medium. By utilizing the SAM2 large model-based air handwriting man-machine interaction method and device, the electronic equipment and the storage medium designed by the invention, the problems that the existing SAM2 large model is insufficient in processing precision and inconvenient to deploy can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and human-computer interaction, and in particular to an air handwriting human-computer interaction method and device based on a SAM2 large model, an electronic device and a storage medium. BACKGROUND

[0002] Non-contact human-computer interaction is an important direction of current technology development. Air handwriting, as a natural interaction method, has application potential in virtual reality, smart home control and other scenarios. However, the existing air handwriting technical solutions have the following limitations:

[0003] The solutions based on depth cameras or inertial sensors are costly or require wearing additional equipment, and the user experience is poor and difficult to popularize. Or the solutions based on ordinary cameras rely on traditional image processing techniques such as background difference and color features, and are easily disturbed by light and complex background, and the accuracy and robustness of hand segmentation are insufficient. In addition, there are also some challenges in the algorithm level: the general deep learning model (such as YOLOv5) has limited detection accuracy for small targets such as fingertips; while the emerging general segmentation large model (such as SAM2) has strong ability, but when directly applied to fine segmentation of hands, the accuracy is insufficient, and the model is large, which is not conducive to terminal device deployment.

[0004] Therefore, the prior art has defects and needs to be improved and developed. SUMMARY

[0005] The present application provides an air handwriting human-computer interaction method and device based on a SAM2 large model, an electronic device and a storage medium, which solves the problems that the existing technology based on depth cameras or inertial sensors is costly or requires wearing additional equipment, and the user experience is poor and difficult to popularize; or the solutions based on ordinary cameras rely on traditional image processing techniques such as background difference and color features, and are easily disturbed by light and complex background, and the accuracy and robustness of hand segmentation are insufficient. In addition, there are also some challenges in the algorithm level: the general deep learning model (such as YOLOv5) has limited detection accuracy for small targets such as fingertips; while the emerging general segmentation large model (such as SAM2) has strong ability, but when directly applied to fine segmentation of hands, the accuracy is insufficient, and the model is large, which is not conducive to terminal device deployment.

[0006] In a first aspect of the present application, an air handwriting human-computer interaction method based on a SAM2 large model is provided, comprising:

[0007] Based on the two-dimensional video stream collected by the monocular camera in real time, a continuous two-dimensional video image sequence is generated after frame processing;

[0008] inputting the two-dimensional video image sequence into the trained SAM2 large model to obtain a hand region segmentation mask;

[0009] extracting a hand contour based on the hand region segmentation mask, positioning a hand centroid according to a geometric structure feature of the hand, and constructing a maximum inscribed circle to separate a palm region and a finger region from the hand contour based on the maximum inscribed circle, and positioning and acquiring coordinates of fingertips in the finger region based on a multi-angle fingertip template matching algorithm;

[0010] When in a writing start state, continuously tracking and recording a coordinate sequence generated in a fingertip movement process, and performing horizontal translation processing on the coordinate sequence based on a coordinate system virtual sliding technology to generate an aerial writing text trajectory without superposition.

[0011] Further, the inputting the two-dimensional video image sequence into the trained SAM2 large model to obtain a hand region segmentation mask includes a training method of the SAM2 large model:

[0012] constructing a special gesture segmentation dataset, the special gesture segmentation dataset containing aerial handwriting original images collected under different backgrounds and illumination conditions and corresponding hand contour annotations;

[0013] using the aerial handwriting original images as inputs of the SAM2 large model, and freezing weights of an image encoder during training;

[0014] using the hand contour annotations as training targets to perform end-to-end supervised training on a mask decoder and a prompt encoder of the SAM2 large model, and randomly sampling 3 foreground points from real hand mask labels as position prompts to input into the prompt encoder of the SAM2 large model during training;

[0015] evaluating performance of the SAM2 large model on a validation set, saving model weights with optimal performance, and obtaining the trained SAM2 large model.

[0016] Further, the extracting a hand contour based on the hand region segmentation mask, positioning a hand centroid according to a geometric structure feature of the hand, and constructing a maximum inscribed circle to separate a palm region and a finger region from the hand contour based on the maximum inscribed circle, and positioning and acquiring coordinates of fingertips in the finger region based on a multi-angle fingertip template matching algorithm includes:

[0017] performing distance transformation on the hand region segmentation mask, and positioning a pixel point coordinate with a maximum distance value as the hand centroid ;

[0018] using the hand centroid The center of the circle is the hand centroid, and the maximum inscribed circle completely contained in the hand region segmentation mask is found, and the radius of the maximum inscribed circle is denoted as ;

[0019] The separation circle is constructed with the hand centroid as the center and the radius of ; , wherein is an amplification coefficient greater than 1;

[0020] All pixel values inside the separation circle are set to zero to obtain a binary image containing only the finger contour;

[0021] The binary image is matched based on a multi-angle fingertip template matching algorithm, and the center point coordinates of the template with the highest matching degree are determined as the fingertip coordinates, and the coordinates of the fingertip are obtained.

[0022] Further, in the method of determining the coordinates of the fingertip by matching the binary image based on the multi-angle fingertip template matching algorithm, the matching degree is calculated by the following formula:

[0023] ; wherein , , represents the horizontal and vertical coordinate values of the upper left corner of the current fingertip template on the binary image of the finger; represents the matching value at the coordinates of the binary image ; , respectively represent the width and height of the fingertip template; , as a loop variable, used to traverse each pixel position in the template, represents the horizontal index loop variable, represents the vertical index loop variable, represents the pixel value of the fingertip template at the coordinates of the binary image ; represents the pixel value of the binary image at coordinates.

[0024] Further, when in the state of starting writing, the coordinate sequence generated in the process of moving the fingertip is continuously tracked and recorded, and the coordinate sequence is processed by horizontal translation based on coordinate system virtual sliding technology to generate a non-overlapping air writing text track, including:

[0025] When in the state of starting writing, the fingertip coordinates obtained in each frame are added to the coordinate sequence;

[0026] In each frame processing, the abscissa of all historical coordinates in the coordinate sequence is uniformly left translated, and the ordinate remains unchanged; wherein the update formula of each coordinate in the coordinate sequence is: ; wherein, represents the coordinate before translation; is the horizontal translation speed; is the time interval; , respectively represent the horizontal and vertical coordinate values after translation;

[0027] Connect all the new coordinates of all historical coordinates after translation to generate a sliding and non-overlapping air writing text track.

[0028] In a second aspect of the embodiment of the application, an air handwritten human-computer interaction device based on a SAM2 large model is provided, comprising:

[0029] The acquisition module is configured to generate a continuous two-dimensional video image sequence based on frame processing on a two-dimensional video stream collected in real time by a monocular camera;

[0030] The input module is configured to input the two-dimensional video image sequence into a trained SAM2 large model to obtain a hand region segmentation mask;

[0031] The processing module is configured to extract a hand contour based on the hand region segmentation mask, locate a hand centroid according to a geometric structure feature of the hand, and construct a maximum inscribed circle to separate a palm and a finger region from the hand contour based on the maximum inscribed circle, and locate and obtain coordinates of a fingertip in the finger region based on a multi-angle fingertip template matching algorithm;

[0032] The output module is configured to continuously track and record a coordinate sequence generated in a fingertip movement process when in a writing start state, perform horizontal translation processing on the coordinate sequence based on coordinate system virtual sliding technology, and generate a non-overlapping air writing text track.

[0033] In a third aspect of the embodiment of the application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the air handwritten human-computer interaction method based on the SAM2 large model.

[0034] In a fourth aspect of the embodiment of the application, a computer readable storage medium is provided, configured to store a computer program, wherein when the computer program is executed on a computer, the computer executes the air handwritten human-computer interaction method based on the SAM2 large model.

[0035] Beneficial effects:

[0036] From the above technical solutions, the present application provides an air handwritten human-computer interaction method based on a SAM2 large model, aiming to solve the problems of inaccurate hand segmentation, high missing detection rate of fingertips, high device cost and trajectory overlap in the prior art, by fine-tuning the mask decoder and prompt encoder of SAM2, combining a self-built gesture dataset, realizing high-precision hand segmentation, and using geometric features and template matching to realize fingertip positioning, and finally generating smooth writing trajectories through coordinate system virtual sliding technology, the present application has the following beneficial effects:

[0037] 1. By fine-tuning the mask decoder and prompt encoder of the SAM2 large model, it focuses on hand features, while retaining the strong generalization ability of the model, significantly improving the hand segmentation accuracy under different scenes and lighting conditions.

[0038] 2. By combining the palm rejection method based on distance transformation and maximum inscribed circle, and the multi-angle template matching strategy that can adapt to changes in finger posture, the missing detection rate of fingertips is effectively reduced, and high-precision and stable positioning under various gestures is realized.

[0039] 3. By using coordinate system virtual sliding technology, when the user writes in a fixed space position, the generated trajectory moves smoothly in one direction on the screen, effectively avoiding the superposition and congestion of writing trajectories, and expanding the effective writing space.

[0040] 4. By using an ordinary monocular camera, such as a built-in camera of a notebook computer or a mobile phone, the hardware cost is low, and it is easy to deploy and promote in various consumer-level electronic devices.

[0041] It should be understood that all combinations of the aforementioned concepts and additional concepts described in greater detail below can be seen as part of the inventive subject matter of the present disclosure, as long as such concepts do not contradict each other.

[0042] The foregoing and other aspects, embodiments and features of the present teachings can be more fully understood from the following description, taken in conjunction with the accompanying drawings. Other aspects, embodiments and features of the present teachings will be apparent from the description that follows, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0043] The drawings are not drawn to true scale. In the drawings, each same or like component is denoted with the same reference numeral. For the sake of clarity, not every component is marked in every drawing. Embodiments of various aspects of the present teachings will now be described, by way of example, with reference to the drawings, in which:

[0044] Figure 1 A total flowchart of a method for air handwriting human-computer interaction based on a SAM2 large model according to an embodiment of the present application.

[0045] Figure 2 A model diagram of gesture segmentation of a method for air handwriting human-computer interaction based on a SAM2 large model according to an embodiment of the present application.

[0046] Figure 3 A schematic diagram of a hand region segmentation and fingertip coordinate positioning process of a method for air handwriting human-computer interaction based on a SAM2 large model according to an embodiment of the present application.

[0047] Figure 4 An air handwriting text trajectory effect diagram of a method for air handwriting human-computer interaction based on a SAM2 large model according to an embodiment of the present application.

[0048] Figure 5 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described below in detail with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the described embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort fall within the scope of the present application. Unless otherwise defined, the technical terms or scientific terms used herein should be understood as their common meanings to those of ordinary skill in the art.

[0050] The terms “first”, “second”, and similar terms used in the patent application specification and claims of the present application do not denote any order, quantity, or importance, but are only used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms “a”, “an”, and “the” and the like do not denote a quantity restriction, but denote the existence of at least one. The terms “include” or “contain” and the like mean that the elements or objects appearing before “include” or “contain” cover the features, whole, steps, operations, elements, and / or components listed after “include” or “contain”, and do not exclude the existence or addition of one or more other features, whole, steps, operations, elements, components, and / or sets thereof. “Up”, “down”, “left”, “right”, and the like are only used to represent relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships may also change accordingly.

[0051] In the prior art, due to the solutions based on depth cameras or inertial sensors, the cost is high or additional equipment needs to be worn, the user experience is poor and it is difficult to popularize; or the solutions based on ordinary cameras rely on traditional image processing technologies such as background difference and color features, and are easily disturbed by light and complex backgrounds, and the accuracy and robustness of hand segmentation are insufficient. In addition, there are also some challenges in the algorithm level: the general deep learning model (such as YOLOv5) has limited detection accuracy for small targets such as fingertips; while the emerging general segmentation large model (such as SAM2) has strong ability, but when directly applied to fine hand segmentation, the accuracy is insufficient, and the model is large, which is not conducive to terminal device deployment.

[0052] Therefore, for this specific task of air handwriting, it is necessary to retain the strong feature extraction capability of SAM2 and the video segmentation on the basis of fine-tuning to adapt to the needs of hand segmentation. By constructing a high-quality gesture dataset and optimizing the mask decoder and prompt encoder of SAM2, the segmentation accuracy of the model for hand contours and details can be improved without excessive increase in computational burden, so as to optimize the segmentation effect.

[0053] In view of this, the embodiment of the present application provides an air handwriting human-computer interaction method based on a SAM2 large model, referring to Figure 1 , comprising:

[0054] Step S102: based on the two-dimensional video stream collected by the monocular camera in real time, a continuous two-dimensional video image sequence is generated. Through a monocular camera, such as a notebook computer built-in camera, an external USB camera or a mobile phone main camera, a real-time two-dimensional video stream is collected at a rate of 30 frames per second. Then, the collected video stream is frame extracted and decoded into a continuous two-dimensional video image sequence for subsequent processing.

[0055] Step S104: input the two-dimensional video image sequence into the trained SAM2 large model to obtain a hand region segmentation mask. The current frame image obtained is input into a SAM2 large model trained by fine-tuning, and a binary hand region segmentation mask with the same resolution as the input image is output by the model. In the mask, the foreground (hand region) pixel value is 255, and the background pixel value is 0.

[0056] Step S106: extracting the hand contour based on the hand region segmentation mask, positioning the hand centroid according to the geometric structure characteristics of the hand and constructing the largest inscribed circle to separate the palm and finger regions from the hand contour based on the largest inscribed circle, and positioning and obtaining the coordinates of the fingertips in the finger region based on the multi-angle fingertip template matching algorithm.

[0057] Step S108: When in the state of writing start, the coordinate sequence generated in the process of the fingertip movement is continuously tracked and recorded, and the coordinate sequence is processed by horizontal translation based on the coordinate system virtual sliding technology, to generate an aerial writing text track without superposition.

[0058] In some embodiments, a two-dimensional video image sequence is input into the trained SAM2 large model, and a hand region segmentation mask is obtained. The training method of the SAM2 large model includes:

[0059] A special gesture segmentation dataset is constructed, which includes aerial handwriting original images collected under different background and lighting conditions and corresponding hand contour annotations.

[0060] The aerial handwriting original image is used as the input of the SAM2 large model, and the weights of the image encoder are frozen during the training process.

[0061] The hand contour annotation is used as the training target to perform end-to-end supervised training on the mask decoder and prompt encoder of the SAM2 large model. During the training process, 3 foreground points are randomly sampled from the real hand mask label as position prompt input to the prompt encoder of the SAM2 large model.

[0062] The performance of the SAM2 large model is evaluated on the validation set, and the model weight with the best performance is saved to obtain the trained SAM2 large model.

[0063] The specific training method of the SAM2 large model is as follows:

[0064] Under various background and lighting conditions such as indoor, outdoor, strong light, and weak light, a monocular camera is used to collect multiple videos for aerial writing. Video frame images are extracted, and the hand contour in each frame image is labeled using an image labeling tool to generate corresponding binary mask labels, thereby constructing a special gesture segmentation dataset. The dataset finally contains about 13144 images, and is randomly divided into a training set and a test set in a ratio of 8:2.

[0065] Reference Figure 2 The pre-training weights of the SAM2 model are loaded using the PyTorch deep learning framework. During the fine-tuning training process, the image encoder serves as a general visual feature extractor and has strong cross-domain representation ability after pre-training on a large amount of data. The parameters of the image encoder are frozen to maintain its general visual features, and only the parameters of the prompt encoder and the mask decoder are updated.

[0066] During training, input images are uniformly scaled to 512x512 pixels. To simulate the case that rough hand position information is provided in real interaction, 3 foreground points are randomly sampled from the real hand mask label as position cue input to the model during training, forcing the model to learn the ability to infer the complete object shape from a single vague cue.

[0067] To improve both segmentation accuracy and output reliability, a compound loss function is used, containing two parts: segmentation loss and confidence calibration loss. The segmentation loss adopts the standard binary cross-entropy function, while the confidence calibration loss measures the difference between the model's predicted confidence score and the true IoU through L1 norm. The final loss is the weighted sum of these two losses, balancing between segmentation accuracy and confidence calibration. The AdamW optimizer is used for parameter update, with an initial learning rate of 1e-5, a Batch Size of 16, and a total of 200 Epochs of training.

[0068] During training, the model performance is continuously evaluated on the validation set, with the mean intersection over union mIoU as the evaluation metric. When the validation set mIoU no longer improves for 10 consecutive Epochs, training is terminated early, and the model weight file with the best performance is saved for subsequent system deployment and inference.

[0069] In some embodiments, referring to Figure 3 , the hand contour is extracted based on the hand region segmentation mask, the hand centroid is located according to the geometric structure features of the hand, and the largest inscribed circle is constructed to separate the palm and finger regions from the hand contour based on the largest inscribed circle, and the coordinates of the fingertips are located and obtained in the finger region based on a multi-angle fingertip template matching algorithm, including:

[0070] The hand region segmentation mask is subjected to distance transformation, and the pixel point coordinates with the largest distance value are located as the hand centroid ;

[0071] The largest inscribed circle that can be completely contained in the hand region segmentation mask is found with the hand centroid as the center, and the radius of the largest inscribed circle is denoted as ;

[0072] The separation circle is constructed with the hand centroid as the center and as the radius, where is an amplification coefficient with a value greater than 1;

[0073] All pixel values inside the separation circle are set to zero, obtaining a binary image containing only the finger contour;

[0074] Based on multi-angle fingertip template matching algorithm matches binary image, the highest matching degree template center point coordinates is determined as the fingertip coordinates, and then the fingertip coordinates are obtained.

[0075] The specific steps of obtaining the fingertip coordinates are as follows:

[0076] (1) The distance transform is performed on the obtained hand region segmentation mask. The Euclidean distance of each foreground pixel point in the mask to its nearest background pixel point is calculated. Then, the pixel point with the maximum distance value is found, and the coordinates of the point are located as the hand center . The center point is usually located in the palm center region.

[0077] (2) Taking the center as the center, find the largest inscribed circle completely contained in the hand mask, and the radius of the circle is . According to a large number of experimental verification, set an amplification coefficient , the value is 1.5. Then, taking the same center as the center, and as the radius, draw a circle, which is the separation circle. Set the value of all pixel points inside the separation circle to 0, so as to remove the palm region in the original hand mask, and finally obtain a binary image containing only the finger contour.

[0078] (3) Multi-angle fingertip template matching. An initial fingertip template image is designed in advance, which is a binary image with a size of 20x20 pixels and pointing upwards. The initial template is rotated every 5 degrees in clockwise and counterclockwise directions, 8 times in each direction, resulting in 17 fingertip templates of different angles, forming a multi-angle fingertip template library.

[0079] (4) Each fingertip template in the multi-angle fingertip template library is traversed on the obtained finger binary image from the top left corner with a step of 1 pixel. Through matching degree calculation, the center point coordinates of the template with the highest matching degree are determined as the fingertip coordinates, and then the fingertip coordinates are obtained.

[0080] In some embodiments, based on multi-angle fingertip template matching algorithm matches binary image, the highest matching degree template center point coordinates is determined as the fingertip coordinates, and then the fingertip coordinates are obtained, and the matching degree calculation method includes:

[0081] The matching degree is calculated according to the following formula: ; wherein, , represents the left upper corner horizontal and vertical coordinate values of the current fingertip template on the finger binary image; represents the matching value at the coordinates of the binary image ; , width and height of the fingertip template, respectively; , as a loop variable for iterating through each pixel position within the template, as a loop variable for iterating through the horizontal index, as a loop variable for iterating through the vertical index, represents the pixel value of the fingertip template at the binary image coordinate, represents the pixel value of the binary image at the coordinate.

[0082] After iterating through all the templates and all the positions, the globally minimum matching value and its corresponding template position and template size are found. The center point coordinate of the optimal matching template is taken as the fingertip coordinate of the current frame, and the calculation formula is: , .

[0083] In some embodiments, when in the state of writing start, the coordinate sequence generated in the process of fingertip movement is continuously tracked and recorded, and the coordinate sequence is processed for horizontal translation based on the coordinate system virtual sliding technology to generate a non-overlapped air writing text trajectory, including:

[0084] When in the state of writing start, the fingertip coordinate obtained in each frame is added to the coordinate sequence;

[0085] In each frame processing, the horizontal coordinates of all historical coordinates in the coordinate sequence are uniformly translated leftward, and the vertical coordinates remain unchanged; wherein, the update formula of each coordinate in the coordinate sequence is: ; wherein, represents the coordinate before translation; is the horizontal translation speed; is the time interval; , represent the horizontal and vertical coordinate values after translation, respectively;

[0086] The new coordinates after translation of all historical coordinates are connected to generate a sliding and non-overlapped air writing text trajectory.

[0087] The specific steps for generating a sliding and non-overlapped air writing text trajectory are as follows:

[0088] First, an empty trajectory coordinate queue is initialized. When the user enters the state of writing start, the fingertip coordinate located in each frame is started to be added to the queue .

[0089] Meanwhile, to address the issue of overlapping writing trajectories in long texts, a virtual coordinate system sliding technique is applied during each frame processing. This technique applies to queues. The x-coordinates of all stored historical coordinates are uniformly shifted to the left, while the y-coordinates remain unchanged. Specifically, this is done through each coordinate... The update formula updates each coordinate in the queue in real time, where the lateral movement speed... Set to 2 pixels per frame.

[0090] Ultimately, the aerial handwritten text trajectory used for display on the screen and subsequent recognition is formed by connecting a new coordinate sequence after translation processing.

[0091] like Figure 4 As shown, this technology simulates the reverse sliding of the coordinate system through a program. When a user writes in a fixed position in the air, the trajectory will appear as a smooth movement from right to left on the screen, thus leaving sufficient virtual space for subsequent written characters and fundamentally avoiding the problems of character superposition and distortion caused by the fixed field of view of the camera.

[0092] Another embodiment of the present invention also provides an aerial handwriting human-computer interaction device based on the SAM2 large model, comprising:

[0093] The acquisition module is used to generate a continuous sequence of two-dimensional video images by performing frame-by-frame processing on the two-dimensional video stream acquired in real time by the monocular camera.

[0094] The input module is used to input a sequence of two-dimensional video images into the trained SAM2 model to obtain a hand region segmentation mask.

[0095] The processing module is used to extract the hand contour based on the hand region segmentation mask, locate the hand centroid based on the geometric features of the hand and construct the maximum inscribed circle to separate the palm and finger regions from the hand contour based on the maximum inscribed circle, and locate and obtain the coordinates of the fingertips in the finger region based on the multi-angle fingertip template matching algorithm.

[0096] The output module is used to continuously track and record the coordinate sequence generated during the fingertip movement when the writing state begins. Based on the coordinate system virtual sliding technology, the coordinate sequence is translated laterally to generate a non-overlapping aerial writing text trajectory.

[0097] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0098] Based on the same inventive concept as the method embodiments, the electronic device in the embodiments of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the electronic device implements the control method in the above embodiments.

[0099] In an embodiment, the electronic device can be a server, and in this embodiment, the structure of the electronic device can be as shown in Figure 5 , including a memory, a communication module, and one or more processors.

[0100] The memory is used to store the computer program executed by the processor. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and programs required for running instant messaging functions, etc.; and the data storage area can store various instant messaging information and operation instruction sets, etc.

[0101] The memory can be a volatile memory, such as a random access memory (RAM); the memory can also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid state drive (SSD); or the memory can be any other medium capable of carrying or storing a desired computer program in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory can be a combination of the above memories.

[0102] The processor can include one or more central processing units (CPUs) or digital processing units, etc. The processor is used to call the computer program stored in the memory to implement the above-mentioned audio data processing method.

[0103] The communication module is used to communicate with the terminal device and other servers.

[0104] The specific connection medium between the above-mentioned memory, communication module and processor is not limited in the embodiments of the present application. In the embodiments of the present application, the memory and the processor are connected through a bus, Figure 5 and the bus is described by an arrow in Figure 5 , and the connection mode between other components is only schematically described and is not limited thereto. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of description, Figure 5Only one arrow is used to represent a single bus or line unless otherwise specified.

[0105] With the same inventive concept as the above method embodiments, the embodiments of the present application also provide a computer readable storage medium, which is used to store a computer program, and when the computer program is run on a computer, the computer program causes an electronic device to implement the control method in the above embodiments. The computer readable storage medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0106] With the same inventive concept as the above method embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program, and when the program product is run on an electronic device, the computer program is used to cause the electronic device to perform the steps in the control method according to various exemplary embodiments of the present application described in the specification. The program product can adopt any combination of one or more readable media. These computer program commands can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the commands executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in the flowchart Figure 1 The device for implementing the functions specified in one or more flows or one or more blocks. Figure 1 The device for implementing the functions specified in one or more flows or one or more blocks.

[0107] Although the present application has been disclosed in the above preferred embodiments, it is not intended to limit the present application. Those skilled in the art who have ordinary knowledge can make various modifications and improvements without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application should be defined by the claims.

Claims

1. A method for aerial handwriting human-computer interaction based on the SAM2 large model, characterized in that, include: A continuous sequence of two-dimensional video images is generated by processing frames of the two-dimensional video stream captured in real time by a monocular camera. Input the two-dimensional video image sequence into the trained SAM2 large model to obtain the hand region segmentation mask; The hand contour is extracted based on the hand region segmentation mask. The centroid of the hand is located according to the geometric features of the hand, and the maximum inscribed circle is constructed to separate the palm and finger regions from the hand contour based on the maximum inscribed circle. The coordinates of the fingertips are located and obtained in the finger region based on the multi-angle fingertip template matching algorithm. When the writing begins, the coordinate sequence generated during the fingertip movement is continuously tracked and recorded. The coordinate sequence is then horizontally translated based on the coordinate system virtual sliding technology to generate a non-overlapping aerial writing text trajectory.

2. The aerial handwriting human-computer interaction method based on the SAM2 large model according to claim 1, characterized in that, The process of inputting the two-dimensional video image sequence into the trained SAM2 large model to obtain the hand region segmentation mask includes the following training methods for the SAM2 large model: A dedicated gesture segmentation dataset is constructed, which includes original images of aerial handwriting collected under different background and lighting conditions and corresponding hand contour annotations. The original handwritten image in the air was used as the input to the SAM2 large model, and the weights of the image encoder were frozen during training. Using the hand contour annotation as the training target, end-to-end supervised training is performed on the mask decoder and cue encoder of the SAM2 large model. During the training process, three foreground points are randomly sampled from the real hand mask labels as position cue inputs to the cue encoder of the SAM2 large model. The performance of the SAM2 large model is evaluated on the validation set, and the weights of the best-performing model are saved to obtain the trained SAM2 large model.

3. The aerial handwriting human-computer interaction method based on the SAM2 large model according to claim 1, characterized in that, The process of extracting the hand contour based on the hand region segmentation mask, locating the hand centroid based on the geometric features of the hand and constructing a maximum inscribed circle to separate the palm and finger regions from the hand contour based on the maximum inscribed circle, and locating and obtaining the coordinates of the fingertips in the finger region based on a multi-angle fingertip template matching algorithm, includes: A distance transformation is performed on the hand region segmentation mask, and the coordinates of the pixel with the maximum distance value are located as the centroid of the hand. ; With the aforementioned hand center of mass Using the circle as the center, find the largest inscribed circle that is completely contained within the hand region segmentation mask, and denoted by the radius of the largest inscribed circle. ; With the aforementioned hand center of mass With the center as the center, Construct a separating circle with radius, where, Amplification factor with a value greater than 1; Set all pixel values ​​inside the separation circle to zero to obtain a binary image containing only the finger outline; The binary image is matched based on a multi-angle fingertip template matching algorithm. The coordinates of the center point of the template with the highest matching degree are determined as the fingertip coordinates, thereby obtaining the fingertip coordinates.

4. The aerial handwriting human-computer interaction method based on the SAM2 large model according to claim 3, characterized in that, The multi-angle fingertip template matching algorithm matches the binary image, and determines the center point coordinates of the template with the highest matching degree as the fingertip coordinates, thereby obtaining the fingertip coordinates. The method for calculating the matching degree includes: The matching degree is calculated using the following formula: ;in, , This represents the horizontal and vertical coordinates of the top-left corner of the current fingertip template in the binary image of the finger; In the binary image Matching value at coordinates; , These represent the width and height of the fingertip template, respectively. , It serves as a loop variable, used to iterate through each pixel position within the template. The loop variable representing the horizontal index. The loop variable representing the vertical index. The fingertip template is represented in the binary image Pixel value at coordinates The binary image represents the image in The pixel value at the coordinates.

5. The aerial handwriting human-computer interaction method based on the SAM2 large model according to claim 1, characterized in that, When the writing begins, the coordinate sequence generated during fingertip movement is continuously tracked and recorded. Based on virtual sliding technology of the coordinate system, the coordinate sequence is horizontally translated to generate a non-overlapping aerial writing text trajectory, including: When writing begins, the fingertip coordinates acquired in each frame are added to the coordinate sequence; During each frame processing, the x-coordinates of all historical coordinates in the coordinate sequence are uniformly shifted to the left, while the y-coordinates remain unchanged; wherein, for each coordinate in the coordinate sequence... The update formula is: ;in, Indicates the coordinates before translation; It is the lateral speed; For time intervals; , These represent the horizontal and vertical coordinates after translation; Connect all the historical coordinates after translation to generate a sliding, non-overlapping aerial text trajectory.

6. An aerial handwriting human-computer interaction device based on the SAM2 large model, characterized in that, include: The acquisition module is used to generate a continuous sequence of two-dimensional video images by performing frame-by-frame processing on the two-dimensional video stream acquired in real time by the monocular camera. The input module is used to input the two-dimensional video image sequence into the trained SAM2 large model to obtain the hand region segmentation mask; The processing module is used to extract the hand contour based on the hand region segmentation mask, locate the hand centroid according to the geometric structural features of the hand and construct the maximum inscribed circle to separate the palm and finger regions from the hand contour based on the maximum inscribed circle, and locate and obtain the coordinates of the fingertips in the finger region based on the multi-angle fingertip template matching algorithm. The output module is used to continuously track and record the coordinate sequence generated during the fingertip movement when the writing state is started, and to perform lateral translation processing on the coordinate sequence based on the coordinate system virtual sliding technology to generate a non-overlapping aerial writing text trajectory.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it enables the electronic device to implement the aerial handwriting human-computer interaction method based on the SAM2 large model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer, causes the computer to perform the aerial handwriting human-computer interaction method based on the SAM2 large model as described in any one of claims 1 to 5.

Images