Three-dimensional handwriting generation method, device and system

Abstract

The present application relates to a kind of three-dimensional handwriting generation method, equipment and system, belong to human-computer interaction technical field, this three-dimensional handwriting generation method, equipment and system, can be real-time drawing and writing without any specific hardware equipment, low in cost;Pen tip detection model is pre-constructed by depth learning model, to detect pen tip, and the pen tip position is mapped to three-dimensional space, obtains three-dimensional space coordinate information, to track the coordinate information of three-dimensional space in real time, obtains predicted three-dimensional coordinates, and the three-dimensional coordinate graph of handwriting movement is drawn, error is small, effect is better, with universality.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to a method, device, and system for generating three-dimensional handwriting. Background Technology

[0002] Human-Computer Interaction (HCI) technology, based on computer technology, establishes "communication" between humans and machines. HCI technology involves sending instructions to machines through specific human-computer interaction actions and procedures, enabling the machines to respond with feedback based on the received instructions, thus adapting to human needs. Among various HCI devices, the HCI pen is a crucial medium for realizing human-machine interaction and feedback.

[0003] Traditional human-computer interaction pens typically employ numerous inertial sensors for information acquisition. However, a large number of sensors are often expensive; moreover, various noises and external interferences exist during operation, causing errors during interaction and reducing the accuracy of handwriting generation.

[0004] Therefore, existing technologies suffer from high costs and low accuracy in acquiring handwriting using human-computer interaction pens. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a three-dimensional handwriting generation method, device and system to overcome the technical problems of high cost and low accuracy in human-computer interaction pen handwriting acquisition.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] On the one hand, a method for generating three-dimensional handwriting includes:

[0008] The operation image is obtained by acquiring the user's hand holding an interactive pen and performing pen gesture interaction based on a depth camera. The operation image includes the hand and the interactive pen. The operation image is a color image captured by the RGB color camera in the depth camera. The color image is a two-dimensional image.

[0009] A hand detection box is determined in the operation image based on the hand enlargement rule, wherein the hand detection box includes the complete interactive pen;

[0010] The position of the pen tip is determined within the hand detection frame according to the preset pen tip detection model;

[0011] The position of the pen tip is located in three-dimensional space to obtain the coordinate information of the pen tip in three-dimensional space;

[0012] The tracking algorithm based on the 3D Kalman filter tracks the coordinate information of the pen tip in three-dimensional space in real time to obtain the predicted three-dimensional coordinates.

[0013] Based on the predicted coordinates, a three-dimensional coordinate graph of the handwriting movement is drawn using 3D software.

[0014] Optionally, determining the position of the pen tip within the hand detection frame based on a preset pen tip detection model includes:

[0015] Image features are extracted from the hand detection frame based on the preset pen tip detection model. The bounding box information of the pen tip within the hand detection frame is calculated based on these image features. The bounding box and its contents are used as the pen tip position. The bounding box information includes U-coordinates u. i V coordinate v i The width w of the pen tip bounding box i and height h i .

[0016] Optionally, the step of locating the position of the pen tip in three-dimensional space to obtain the coordinate information of the pen tip in three-dimensional space includes:

[0017] Based on the pen tip position, determine the two-dimensional coordinates of the pen tip's centroid.

[0018] Obtain depth information at the pixel point corresponding to the two-dimensional coordinates of the pen tip centroid in the depth map;

[0019] The coordinates of the pen tip in three-dimensional space are obtained based on the depth information.

[0020] Optional, also includes:

[0021] Align the depth map data acquired from the depth camera with the color image data.

[0022] Optionally, aligning the depth map data acquired from the depth camera with the color image data includes:

[0023] Convert 2D points in the depth map to 3D points in the world coordinate system;

[0024] Project 3D points in the world coordinate system onto the color image.

[0025] Optionally, the tracking algorithm based on a 3D Kalman filter tracks the coordinate information of the pen tip in three-dimensional space in real time to obtain predicted three-dimensional coordinates, including:

[0026] Determine the three-dimensional coordinates of the pen tip in the current operation image and the three-dimensional coordinates of the pen tip in the previous operation image;

[0027] Based on a 3D Kalman filter-based tracking algorithm, the three-dimensional coordinates of the pen tip in the current operation image and the three-dimensional coordinates of the pen tip in the previous operation image are used to predict the trajectory state of the pen tip in three-dimensional space from the previous operation image to the current operation image, thus obtaining the predicted three-dimensional coordinates.

[0028] Optionally, the step of using 3D software to draw a three-dimensional coordinate map of the handwriting movement based on the predicted coordinates includes:

[0029] Based on the coordinate information of the pen tip in three-dimensional space and the predicted coordinates, a three-dimensional coordinate graph of the pen movement is drawn using 3D software.

[0030] Optionally, the preset pen tip detection model includes a YOLOv5 convolutional neural network model, wherein the YOLOv5 convolutional neural network model is obtained after training based on interactive pen sample images and pre-determined interactive pen tip labels.

[0031] In another aspect, a three-dimensional handwriting generation device includes a processor and a memory, wherein the processor is connected to the memory:

[0032] The processor is used to call and execute the program stored in the memory;

[0033] The memory is used to store the program, which is at least used to execute the three-dimensional handwriting generation method according to any one of claims 1-8.

[0034] On the other hand, a three-dimensional handwriting generation system includes: a depth camera and the aforementioned three-dimensional handwriting generation device connected to each other;

[0035] The depth camera is used to capture images of the interactive pen at a preset distance.

[0036] The present invention includes at least the following beneficial effects:

[0037] The three-dimensional handwriting generation method, device, and system provided in this invention are based on acquiring operation images of a user holding an interactive pen and performing pen gesture interactions using a depth camera. The operation images include the hand and the interactive pen. The operation images are color images captured by an RGB color camera in the depth camera, and the color images are two-dimensional images. A hand detection box is determined in the operation image based on hand enlargement rules, wherein the hand detection box includes the complete interactive pen. The position of the pen tip is determined within the hand detection box according to a preset pen tip detection model. The position of the pen tip is located in three-dimensional space to obtain the coordinate information of the pen tip in three-dimensional space. A tracking algorithm based on a 3D Kalman filter is used to track the coordinate information of the pen tip in three-dimensional space in real time to obtain predicted three-dimensional coordinates. Based on the predicted coordinates, a three-dimensional coordinate map of the handwriting movement is drawn using 3D software. Therefore, the technical solution of this application does not require any specific hardware equipment, and can draw and write in real time at a low cost. By pre-constructing a pen tip detection model through a deep learning model, the pen tip is detected and its position is mapped to a three-dimensional space to obtain three-dimensional spatial coordinate information. This allows for real-time tracking of the three-dimensional spatial coordinate information, obtaining predicted three-dimensional coordinates, and drawing a three-dimensional coordinate graph of the handwriting movement. The error is small, the effect is good, and it has universality. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a three-dimensional handwriting generation method provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of a human-computer interaction system provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the enlarged process of a hand detection frame provided in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of a pen tip detection process provided in an embodiment of the present invention;

[0043] Figure 5 This invention provides a schematic diagram of pen tip imaging in pixel coordinates for an embodiment of the invention;

[0044] Figure 6 This invention provides a projection of a pen tip in a pixel coordinate system.

[0045] Figure 7 This is a schematic diagram of Kalman filter trajectory prediction provided in an embodiment of the present invention;

[0046] Figure 8 This invention provides a scatter plot of the actual and predicted positions of the centroids in different directions, as provided in an embodiment of the invention. Figure 8 (a) is a scatter plot of the actual and predicted values ​​of the centroid location. Figure 8 (b) is a scatter plot of the actual and predicted values ​​of the X-axis coordinates. Figure 8 (c) is a scatter plot of the actual and predicted values ​​of the Y-axis coordinate. Figure 8 (d) is a scatter plot of the actual and predicted values ​​of the Z-direction coordinates and the predicted values.

[0047] Figure 9 This invention provides a three-dimensional handwriting generation device. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0049] With the rapid development of human-computer interaction technology, interactive pen technology has brought new interfaces to human-computer interaction and expanded new human-computer interaction functions. Researchers have also put forward new requirements and directions for improvement for this technology. With the support of various new technologies such as object detection, human pose estimation, and object tracking, it is necessary to design a method that is more suitable for arbitrary pen interaction.

[0050] The following are some current implementations of 3D interactive pen technology:

[0051] Haruki Shima has developed a system using haptic devices to help users master drawing skills when drawing in 3D space. Haptic devices provide force feedback to the user; when a virtual pen touches a virtual 3D object, the learner experiences tactile changes, preventing mistakes while drawing on the virtual object. This method utilizes numerous sensors, and while the haptic feedback is effective, its widespread adoption is limited by the high cost of VR devices.

[0052] InfiniteZ's interactive pen utilizes virtual reality and real-time human-computer interaction technologies to develop tablet computers that support 3D stereoscopic scenes and human-computer interaction. Users can use the infrared interactive pen to perform various operations on the 3D models displayed on the tablet, such as rotation, translation, and scaling.

[0053] In 2019, Logitech released a VR pen called the "Logitech VR Ink Pilot Edition." This was the world's first VR pen to utilize two-dimensional surfaces and aerial views in three-dimensional space, designed to help professionals complete designs more easily in VR. The stylus is manufactured using SteamVR tracking technology, similar to typical SteamVR controllers, but offering natural precision based on finger control.

[0054] The aforementioned sensor-based contactless human-computer interaction pen aligns with people's daily usage behaviors and can perform various operations on complex virtual objects. However, this type of sensor-based contactless interaction pen requires a large number of sensors, making it expensive and only suitable for single scenarios, lacking universality.

[0055] Traditional human-computer interaction pens typically employ numerous inertial sensors for information acquisition. However, a large number of sensors are often expensive; moreover, various noises and external interferences exist during operation, causing errors during interaction and reducing the accuracy of handwriting generation.

[0056] Therefore, existing technologies suffer from high costs and low accuracy in acquiring handwriting using human-computer interaction pens.

[0057] Based on this, embodiments of the present invention provide a method, device and system for generating three-dimensional handwriting.

[0058] Figure 1 This is a flowchart illustrating a three-dimensional handwriting generation method provided in an embodiment of the present invention. The method provided in this application can be executed by a control component, which can be a microcontroller, a programmable controller, etc. (See reference...) Figure 1 This embodiment may include the following steps:

[0059] Step S1: Acquire operation images of the user holding an interactive pen and performing pen gesture interaction based on the depth camera. The operation images include the hand and the interactive pen. The operation images are color images captured by the RGB color camera in the depth camera, and the color images are two-dimensional images.

[0060] In a specific implementation process, the three-dimensional handwriting generation method provided in this application can be applied in a human-computer interaction system.

[0061] Figure 2 This is a schematic diagram of the structure of a human-computer interaction system provided in an embodiment of the present invention. (See attached diagram.) Figure 2 The system provided in this application may include: an interactive pen 21, a depth camera 22, and a 3D handwriting generation device 23; wherein the depth camera and the 3D handwriting generation device are connected, and the depth camera is used to capture operation images of the human-computer interactive pen at a preset distance. The operation images can be frame images, and multiple frame images can form a video.

[0062] It is worth noting that, in order to ensure the accuracy of frame image acquisition, when applying the technical solution described in the embodiments of this application, the user needs to stand at least a preset distance (e.g., 1 meter) from the depth camera so that the depth camera can capture the user's hand posture and the human-computer interaction pen in their hand. The depth camera can be set up independently or installed on a laptop, desktop computer, mobile phone, etc.

[0063] During 3D handwriting generation, a depth camera (e.g., a webcam) can capture the current operation image as a frame image in real time. After the depth camera captures the current frame image, the 3D handwriting generation device can acquire the current frame image.

[0064] In the process, users can hold the interactive pen to draw and perform pen gesture interaction, which allows the depth camera to capture images of the user holding the interactive pen and performing pen gesture interaction.

[0065] The depth camera can be a RealSense D435 depth camera, which mainly consists of an infrared dot projector, a left imager (left depth camera), a right imager (right depth camera), and an RGB color camera. This camera is based on the principle of triangulation. It uses an infrared laser to project light with specific structural features onto the object being photographed. The receiver receives different structured light images from the object at different distances, and finally calculates the depth information according to the triangulation principle. The specific working principle of the depth camera is detailed in existing technology and will not be elaborated upon in this application.

[0066] In specific gesture-based interaction scenarios, the images captured by the depth camera can include color images captured by the RGB color camera and depth images captured by the left and right imagers. The captured two-dimensional color images are used as the operation images.

[0067] Step S2: Determine the hand detection box in the operation image based on the hand enlargement rule, wherein the hand detection box includes the complete interactive pen.

[0068] In this application, after obtaining the operation image, the operation image can be input into the hand detection model, the original hand detection box is determined in the current frame operation image, and the original hand detection box is expanded according to the hand expansion algorithm until the size of the interactive pen in the hand can be fully detected, thus obtaining the final hand detection box.

[0069] It is worth noting that the hand detection model includes the MediaPipe hand detection model (e.g., based on the MediaPipe gesture algorithm). This invention uses the MediaPipe framework as the human pose detection algorithm. Based on the hand key point data generated by MediaPipe, a coordinate scaling algorithm is used to enlarge the hand detection box to a size large enough to fully detect the interactive pen in the hand.

[0070] To improve the accuracy of interactive pen detection, this application uses the MediaPipe hand detection model to obtain the hand detection box. After obtaining the user's hand detection box in the current operation image, the detection range of the interactive pen can be adapted to the area around the user's hand, removing unnecessary interference and improving the detection accuracy of the interactive pen.

[0071] For example, hand skeletal point recognition can be performed using the MediaPipe gesture algorithm, accurately obtaining hand position information from a single image. Real-time detection is supported, and 21 coordinates of the user's hand and finger joints can be obtained. This invention uses a coordinate magnification algorithm to enlarge the hand detection box based on the hand keypoint data generated by MediaPipe, expanding it to a size large enough to completely detect the interactive pen in the hand. Specifically, the coordinates of 21 key points of the hand are obtained and placed in a list. The minimum and maximum values ​​in the x-axis and y-axis directions are calculated from this list. The point with the minimum x-axis value is moved to the left of the x-axis, and the point with the maximum y-axis value is moved to the top of the y-axis, resulting in an enlarged detection box. This enlarged detection box is used as the user's hand detection box. To ensure the magnification effect, preset values ​​can be adjusted to ensure that the enlarged user hand detection box completely includes the hand and pen.

[0072] Figure 3 This is a schematic diagram of the enlargement process of a hand detection box provided in an embodiment of the present invention. The coordinate image is the original hand detection box generated by MediaPipe, and the right side of the arrow is the enlarged hand detection box. The enlarged user hand detection box can completely include the hand and pen.

[0073] This application provides an example of a coordinate scaling algorithm:

[0074] Algorithm: Hand Coordinate Enlargement Algorithm

[0075]

[0076]

[0077] Step S3: Determine the position of the pen tip within the hand detection frame according to the preset pen tip detection model.

[0078] The preset pen tip detection model includes a YOLOv5 convolutional neural network model, which is obtained by training based on interactive pen sample images and pre-determined interactive pen tip labels.

[0079] For example, the training process may include: taking a RealSense D435 depth camera as an example, capturing color images from multiple angles, in multiple scenes, and under multiple lighting conditions in a gesture-based interaction scenario, using these as raw images. The raw images undergo data cleaning and preprocessing operations such as low-light enhancement. The processed images are then labeled using an annotation tool, and the labeled images are randomly divided into training, validation, and test sets at a fixed ratio. YOLOv5 is used to train the dataset, resulting in a trained model, which serves as the preset pen tip detection model.

[0080] Figure 4 This is a schematic diagram of a pen tip detection process provided in an embodiment of the present invention. (See attached diagram.) Figure 4 In this application, the detection of the interactive pen tip is performed within a given hand detection box generated by MediaPipe within the acquired hand detection box to determine whether it is a pen tip. After generating an expanded detection box based on the hand key points of MediaPipe, YOLOv5 is used to detect the existence of the interactive pen and the pen tip position coordinates. The obtained pen tip position is the two-dimensional planar coordinate of the pen tip in the color image.

[0081] In some embodiments, determining the position of the pen tip within a hand detection frame based on a preset pen tip detection model includes:

[0082] Image features are extracted from the hand detection bounding box based on a pre-defined pen tip detection model. The bounding box information of the pen tip within the hand detection bounding box is calculated based on these features. The bounding box and its contents are used as the pen tip position. The bounding box information includes the U-coordinate u. i V coordinate v i The width w of the pen tip bounding box i and height h i .

[0083] After acquiring depth information and two-dimensional color images through a depth camera, a localization method is adopted to obtain the three-dimensional spatial information of the fruit by mapping the position of the two-dimensional center point of the pen tip extracted from the YOLOv5 pen tip recognition model in the color image to the depth map.

[0084] First, obtain a dataset with a one-to-one pixel correspondence between the color image and the depth image; then, extract image features from the color image based on the trained YOLOv5 pen tip detection model, and calculate the bounding box (u) of the pen tip in the color image. i v i w i h i ), where u i v i These are the U and V coordinates of the top left corner of the bounding box, respectively, and w i h i These are the width and height of the bounding box, respectively. The center point pixel coordinates (cu) of the bounding box. i ,cv i It can be calculated using formulas (1) and (2):

[0085] cu i =u i +w i / 2(1)

[0086] cv i =v i +h i / 2(2)

[0087] Where i is the pixel number.

[0088] Step S4: Position the pen tip in three-dimensional space to obtain the coordinate information of the pen tip in three-dimensional space.

[0089] In some embodiments, the method further includes aligning depth map data acquired from a depth camera with color image data.

[0090] In some embodiments, aligning depth map data acquired from a depth camera with color image data includes: converting 2D points of the depth map into 3D points in a world coordinate system; and projecting the 3D points in the world coordinate system onto the color image.

[0091] Because the left and right depth cameras and the RGB color camera of the depth camera have different coordinate systems, and the intrinsic parameters and image sizes of the color and depth cameras are not strictly the same, misalignment occurs between the resulting color and depth images. To obtain the accurate depth value corresponding to each pixel in the color image, it is necessary to map each pixel in the RGB color image to the pixel in the depth image. Therefore, before capturing the target, the depth data and color image data are aligned. The alignment principle involves converting 2D points on the depth map into 3D points in the world coordinate system, and then projecting these 3D points onto the color image. After alignment and registration, the depth value corresponding to the centroid of the pen tip can be accurately obtained from the depth map.

[0092] In some embodiments, the position of the pen tip is located in three-dimensional space to obtain the coordinate information of the pen tip in three-dimensional space, including:

[0093] Determine the two-dimensional coordinates of the pen tip's centroid based on the pen tip's position;

[0094] Obtain the depth information of the pixel corresponding to the two-dimensional coordinates of the pen tip centroid in the depth map;

[0095] The coordinates of the pen tip in three-dimensional space are obtained based on the depth information.

[0096] Figure 5 This invention provides a schematic diagram of pen tip imaging in pixel coordinates, as shown in an embodiment of the invention. Figure 6 This invention provides a projection of a pen tip in a pixel coordinate system. Wherein O C X C Y C Z C Let be the camera coordinate system, ouv be the pixel coordinate system, G be the centroid of the pen tip, AB be the diameter, and W be the spatial point corresponding to the projection center of the outer surface closest to the camera. a, b, g, and w are the projection points of A, B, G, and W in the pixel coordinate system, where g and w coincide in the pixel coordinate system. D1, D2, D3, and D4 are the four vertices of the detection box.

[0097] See Figure 5 G is the centroid of the pen tip, AB is the pen tip diameter, and W is the spatial point corresponding to the projection center of the outer surface of the pen tip closest to the camera. The camera coordinate system is O. C X C Y C Z C Ouv is the pixel coordinate system. A plane centered at G and with diameter AB is parallel to the imaging plane. GW is parallel to Z. C Axis. According to the principle of projection, the projections of A, B, G, and W are respectively... Figure 6 In the context of the image, a, b, g, and w are given, where g and w coincide in the pixel coordinate system. The projection of diameter AB into the pixel coordinate system is ab, and the projection of the pen tip centroid and the projection center of the outer surface of the pen tip near the camera is the center g(w) with diameter ab. Using the RealSense camera, the depth value of the spatial point W in the projection of the outer surface of the pen tip can be obtained, denoted by Z. w If we express this as Z, then the actual depth reference value of the geometric center can be approximately expressed as Z. w Based on the center point pixels of the obtained bounding box, the depth value Z in the 3D camera coordinate system corresponding to the center point g(w) is calculated using the depth data acquired by the depth camera. w The calculation method is shown in equation (3).

[0098] Z w =scale * Depth cui,cvi(3)

[0099] Where scale is the scaling factor, which can be preset; Depth cui,cvi The center pixel coordinates of the bounding box (cu) i ,cv i () Corresponding depth value.

[0100] Step S5: The tracking algorithm based on the 3D Kalman filter tracks the coordinate information of the pen tip in three-dimensional space in real time to obtain the predicted three-dimensional coordinates.

[0101] In some embodiments, a tracking algorithm based on a 3D Kalman filter tracks the coordinate information of the pen tip in three-dimensional space in real time to obtain predicted three-dimensional coordinates, including:

[0102] Determine the three-dimensional coordinates of the pen tip in the current operation image and the three-dimensional coordinates of the pen tip in the previous operation image;

[0103] Based on a 3D Kalman filter-based tracking algorithm, the three-dimensional coordinates of the pen tip in the current operation image and the three-dimensional coordinates of the pen tip in the previous operation image are used to predict the trajectory state of the pen tip in three-dimensional space from the previous operation image to the current operation image, thus obtaining the predicted three-dimensional coordinates.

[0104] This invention uses the YOLO algorithm to detect the pen tip, and then uses RealSense to obtain the three-dimensional spatial coordinate information of the pen tip for each frame. After obtaining the three-dimensional coordinate information of the pen tip, the 3D trajectory of the pen tip is rendered and displayed in 3D space.

[0105] Figure 7 This is a schematic diagram of Kalman filter trajectory prediction provided in an embodiment of the present invention. (See attached diagram.) Figure 7 Among them, (A) 3D target detection: The 3D detection module obtains the 3D detection value from 2D-RGBD fusion. t (B) State Prediction: A 3D Kalman filter in the state prediction step will predict the trajectory T. t-1 The state prediction is the current frame t as T est (C) Data Association: Detection D t And predicted trajectory T est (D) State update: The 3D Kalman filter is based on the correlation between frames; M The corresponding matching detection update T in M By matching the state of the trajectory, the final trajectory T is obtained. t .

[0106] After obtaining the depth map and the 2D detector (color map), the two maps are fused (T0) to perform subsequent steps.

[0107] A detailed explanation of (A) 3D object detection:

[0108] In frame t, the output of the 3D detection module is the detected D t Each detection D t It is represented as a tuple (x,y,z,l,w,h,s), which includes the position of the object's center in 3D space (x,y,z), the 3D size of the object (l,w,h), and the confidence score s obtained from the detector YOLOv5.

[0109] (B) 3D Kalman Filter: State Prediction.

[0110] To predict the object's trajectory state from the previous frame to the current frame, this invention uses a constant velocity model independent of camera self-motion to approximate the object's positional shift between frames. Therefore, this invention does not explicitly estimate self-motion, but rather relies on a motion model to adapt to it. This invention represents the object's trajectory state as a 10-dimensional vector T = (x, y, z, l, w, h, s, v). x ,v y ,v z ), where the additional variable v x ,v y ,v z This represents the velocity of an object in 3D space. Δt is the time interval between the previous frame and the current frame. Based on a constant velocity model, in each frame, the velocity T from the previous frame... t-1 The relevant trajectory state will be used as T est Extended to frame t:

[0111] x est =x t-1 +v x,t-1 △t(4)

[0112] y est =y t-1 +v y,t-1 △t(5)

[0113] z est =z t-1 +v z,t-1 △t(6)

[0114] In this application, the motion velocity of each process is updated using the following formula, (v x ,v y ,v z As shown in (7) to (9):

[0115] v x,t =(x t -x t-1 ) / △t(7)

[0116] v y,t =(y t -y t-1 ) / △t(8)

[0117] v z,t =(z t -z t-1 ) / △t(9)

[0118] Ultimately, for trajectory T t-1 The predicted state in frame t is T est =(x est ,yest,z est ,l,w,h,s,v x ,v y ,v z ).

[0119] (C) Data Association:

[0120] To ensure more accurate prediction of pen trajectories, this invention will use the predicted trajectory T est With the detected D t Matching. The output of the data association is as follows:

[0121] T M ={T1,T2,…,T t}(10)

[0122] D M ={D1,D2,…,D t}(11)

[0123] Among them, T M It is the trajectory matched during the detection process, D M It is the actual detected trajectory, and t indicates which frame is matched. For example, T1 and D1 are a matching combination.

[0124] (D) 3D Kalman Filter: State Update.

[0125] In this invention, T est This refers to the predicted trajectory, T t This refers to the final updated trajectory. To address the uncertainty in state prediction, this invention is based on D... M Update T based on the corresponding actual detection. M The state of each trajectory is determined. Finally, following the Bayesian rule, the final associated trajectory T in frame t is obtained. t (Final updated trajectory).

[0126] T t =(x′,y′,z′,l′,w′,h′,s′,v x ′,vy ′,v z ′)(12).

[0127] Where t∈{1,2,…,T}.

[0128] Step S6: Based on the predicted coordinates, use 3D software to draw a three-dimensional coordinate graph of the handwriting movement.

[0129] In some embodiments, based on the predicted coordinates, a three-dimensional coordinate graph of the handwriting movement is drawn using 3D software, including:

[0130] Based on the coordinates of the pen tip in three-dimensional space and the predicted coordinates, a three-dimensional coordinate graph of the pen movement is drawn using 3D software.

[0131] Based on the predicted coordinates, this invention uses Matplotlib as an example to draw a three-dimensional coordinate graph of handwriting movement.

[0132] The three-dimensional handwriting generation method, device, and system provided in this invention are based on acquiring operation images of a user holding an interactive pen and performing pen gesture interactions using a depth camera. The operation images include the hand and the interactive pen. The operation images are color images captured by an RGB color camera in the depth camera, and the color images are two-dimensional images. A hand detection box is determined in the operation image based on hand enlargement rules, wherein the hand detection box includes the complete interactive pen. The position of the pen tip is determined within the hand detection box according to a preset pen tip detection model. The position of the pen tip is located in three-dimensional space to obtain the coordinate information of the pen tip in three-dimensional space. A tracking algorithm based on a 3D Kalman filter is used to track the coordinate information of the pen tip in three-dimensional space in real time to obtain predicted three-dimensional coordinates. Based on the predicted coordinates, a three-dimensional coordinate map of the handwriting movement is drawn using 3D software. Therefore, the technical solution of this application does not require any specific hardware equipment, and can draw and write in real time at a low cost. By pre-constructing a pen tip detection model through a deep learning model, the pen tip is detected and its position is mapped to a three-dimensional space to obtain three-dimensional spatial coordinate information. This allows for real-time tracking of the three-dimensional spatial coordinate information, obtaining predicted three-dimensional coordinates, and drawing a three-dimensional coordinate graph of the handwriting movement. The error is small, the effect is good, and it has universality.

[0133] To illustrate the technical effects of the technical solution provided in this application, a verification embodiment is also provided:

[0134] To test this invention, several evaluation metrics were defined: the mean absolute error (MAE) and mean absolute percentage error (MAPE) between the algorithm's results and the two-dimensional reference values ​​of the pen tip's centroid and the actual depth reference values ​​of the geometric center were used to evaluate the depth information prediction results. The calculation formula is as follows:

[0135]

[0136]

[0137] In the formula, n is the total sample size, yi pred The algorithm predicts the value, yi true These are actual measured values.

[0138] The experiment used 17 datasets, collected at different locations within the same time period and scene. Depth information was successfully acquired in 15 datasets. The failure to acquire two datasets may be due to the pen tip surface being too smooth, preventing the infrared structured light from easily capturing the pen tip's position, thus preventing the imaging sensor from receiving the returned infrared light and thus failing to acquire depth information. Alternatively, outdoor light may have interfered with the camera's infrared structured light emission, leading to reception errors. The detailed calculations of MAE and MAPE in the three directions are shown in Table 1 below.

[0139] Table 1: MAE and MAPE Analysis

[0140]

[0141] Figure 8 This invention provides a scatter plot of the actual and predicted positions of the centroids in different directions, as provided in an embodiment of the invention. Figure 8 (a) is a scatter plot of the actual and predicted values ​​of the centroid location. Figure 8 (b) is a scatter plot of the actual and predicted values ​​of the X-axis coordinates. Figure 8 (c) is a scatter plot of the actual and predicted values ​​of the Y-axis coordinate. Figure 8 (d) shows the actual and predicted scatter plots of the Z-axis coordinates. The actual and predicted values ​​of the pen tip centroid are fitted in the horizontal (X), vertical (Y), and spatial depth (Z) directions, and scatter plots of the actual and predicted positions are created in these three spatial dimensions, as shown below. Figure 8 As shown. Figure 8 From (a) to (c), it can be seen that the predicted values ​​in the X, Y, and Z directions fit the actual values ​​closely to the curves. The actual centroid of the pen tip and the predicted centroid are mapped to the same three-dimensional space, with a high degree of overlap between the actual and predicted points, resulting in more accurate positioning. Testing has shown that this method accurately meets the requirements.

[0142] Based on a general inventive concept, embodiments of the present invention also provide a three-dimensional handwriting generation device.

[0143] Figure 9 A three-dimensional handwriting generation device is provided as an embodiment of the present invention, see below. Figure 9 The device of this application may include a processor 91 and a memory 92, with the processor connected to the memory:

[0144] The processor is used to call and execute programs stored in memory.

[0145] A memory for storing a program, which is at least used to execute any of the above-mentioned three-dimensional handwriting generation methods.

[0146] Based on a general inventive concept, embodiments of the present invention also provide a three-dimensional handwriting generation system. The three-dimensional handwriting generation system provided by embodiments of the present invention includes: a depth camera and the aforementioned three-dimensional handwriting generation device connected to each other; the depth camera is used to capture operation images of an interactive pen at a preset distance.

[0147] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0148] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0149] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0150] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0151] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0152] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0153] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0154] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0155] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for generating three-dimensional handwriting, characterized in that, include: The operation image is obtained by acquiring the user's hand holding an interactive pen and performing pen gesture interaction based on a depth camera. The operation image includes the hand and the interactive pen. The operation image is a color image captured by the RGB color camera in the depth camera. The color image is a two-dimensional image. A hand detection box is determined in the operation image based on the hand enlargement rule, wherein the hand detection box includes the complete interactive pen; The position of the pen tip is determined within the hand detection frame according to the preset pen tip detection model; The position of the pen tip is located in three-dimensional space to obtain the coordinate information of the pen tip in three-dimensional space; The tracking algorithm based on the 3D Kalman filter tracks the coordinate information of the pen tip in three-dimensional space in real time to obtain the predicted three-dimensional coordinates. Based on the predicted three-dimensional coordinates, a three-dimensional coordinate graph of the handwriting movement is drawn using 3D software.

2. The method according to claim 1, characterized in that, Determining the position of the pen tip within the hand detection frame based on a preset pen tip detection model includes: The image features of the hand detection frame are extracted according to the preset pen tip detection model. The bounding box information of the pen tip in the hand detection frame is calculated based on the image features. The bounding box and the area inside the bounding box are taken as the pen tip position. The bounding box information includes U coordinates, V coordinates, width and height of the pen tip bounding box.

3. The method according to claim 2, characterized in that, The step of locating the position of the pen tip in three-dimensional space to obtain the coordinate information of the pen tip in three-dimensional space includes: Based on the pen tip position, determine the two-dimensional coordinates of the pen tip's centroid. Obtain depth information at the pixel point corresponding to the two-dimensional coordinates of the pen tip centroid in the depth map; The coordinates of the pen tip in three-dimensional space are obtained based on the depth information.

4. The method according to claim 3, characterized in that, Also includes: Align the depth map data acquired from the depth camera with the color image data.

5. The method according to claim 4, characterized in that, Aligning the depth map data acquired from the depth camera with the color image data includes: Convert 2D points in the depth map to 3D points in the world coordinate system; Project 3D points in the world coordinate system onto the color image.

6. The method according to claim 1, characterized in that, The tracking algorithm based on a 3D Kalman filter tracks the coordinates of the pen tip in three-dimensional space in real time to obtain predicted three-dimensional coordinates, including: Determine the three-dimensional coordinates of the pen tip in the current operation image and the three-dimensional coordinates of the pen tip in the previous operation image; Based on a 3D Kalman filter-based tracking algorithm, the three-dimensional coordinates of the pen tip in the current operation image and the three-dimensional coordinates of the pen tip in the previous operation image are used to predict the trajectory state of the pen tip in three-dimensional space from the previous operation image to the current operation image, thus obtaining the predicted three-dimensional coordinates.

7. The method according to claim 1, characterized in that, The step of using 3D software to draw a three-dimensional coordinate map of the handwriting movement based on the predicted coordinates includes: Based on the coordinate information of the pen tip in three-dimensional space and the predicted coordinates, a three-dimensional coordinate graph of the pen movement is drawn using 3D software.

8. The method according to claim 1, characterized in that, The preset pen tip detection model includes a YOLOv5 convolutional neural network model, wherein the YOLOv5 convolutional neural network model is obtained by training based on interactive pen sample images and pre-determined interactive pen tip labels.

9. A three-dimensional handwriting generation device, characterized in that, It includes a processor and a memory, wherein the processor is connected to the memory: The processor is used to call and execute the program stored in the memory; The memory is used to store the program, which is at least used to execute the three-dimensional handwriting generation method according to any one of claims 1-8.

10. A three-dimensional handwriting generation system, characterized in that, include: Interconnected depth cameras and the three-dimensional handwriting generation device as described in claim 9; The depth camera is used to capture images of the interactive pen at a preset distance.

Images