Handwriting input system and handwriting input method

Abstract

A handwriting input system and a handwriting input method are provided. The handwriting input method is adapted to a handwriting input system including a camera device, and includes the following steps. An image sequence captured by a camera device is acquired, wherein the image sequence includes a plurality of captured images. A hand keypoint detection model is used to analyze the hand keypoint location data of the captured images. According to the hand keypoint position data of the captured images, a plurality of finger track coordinates of a target finger in the captured images are determined. A handwriting input operation is performed according to the finger trajectory coordinates through a third-party application.

Claims

1. A handwriting input method, applicable to a handwriting input system including a camera device, the method comprising: The method involves: acquiring an image sequence captured by a camera device, wherein the image sequence includes multiple captured images; analyzing hand keypoint position data in the multiple captured images using a hand keypoint detection model; determining multiple finger trajectory coordinates of a target finger in the multiple captured images based on the hand keypoint position data of the multiple captured images; and performing a handwriting input operation based on the multiple finger trajectory coordinates through a third-party application. The steps of acquiring the image sequence captured by the camera device include: providing the image sequence captured by the camera device to a middleware module through a Device Media Foundation Translator (DMFT) module; and transferring the image sequence to a gesture analysis module in the application layer through the middleware module, wherein the gesture analysis module inputs each of the multiple captured images to the hand keypoint detection model.

2. The handwriting input method as described in claim 1, wherein the intermediary software module includes a WebSocket server.

3. The handwriting input method as described in claim 1, wherein the step of acquiring the image sequence captured by the camera device further includes: The image sequence is provided to an application in the application layer through the intermediary software module.

4. The handwriting input method as described in claim 3, wherein the application is the third-party application.

5. The handwriting input method described in claim 4, wherein the third-party application includes a video conferencing program, and the image sequence is configured as a real-time video feed of the video conferencing program.

6. The handwriting input method of claim 1, wherein the step of determining the coordinates of multiple finger trajectories of the target finger in the multiple captured images based on the hand key point position data of the multiple captured images includes: Extract a fingertip key point from the key point location data of the hand in each of the multiple captured images; And obtain the fingertip coordinates of the fingertip key points in each of the multiple captured images.

7. The handwriting input method as described in claim 1, wherein the step of performing the handwriting input operation through the third-party application based on the coordinates of the plurality of finger trajectories includes: A handwritten image is drawn based on the coordinates of the multiple finger trajectories; the handwritten image is then processed by optical character recognition to obtain the input text in the handwritten image; And the input text in the handwritten image can be applied through the third-party application.

8. The handwriting input method as described in claim 1, wherein the step of performing the handwriting input operation through the third-party application based on the coordinates of the plurality of finger trajectories includes: A handwritten image is drawn based on the coordinates of the multiple finger trajectories; and the handwritten image is displayed through the third-party application.

9. The handwriting input method as described in claim 8, wherein the step of displaying the handwriting image through the third-party application includes: The handwritten image is displayed on a virtual whiteboard in a video conference program.

10. A handwriting input system, comprising: A camera device; a storage device for storing multiple instructions; A processor, coupled to the camera device and the storage device, executes the plurality of instructions to: acquire an image sequence captured by the camera device, wherein the image sequence includes multiple captured images; analyze hand keypoint position data in the multiple captured images using a hand keypoint detection model; determine multiple finger trajectory coordinates of a target finger in the multiple captured images based on the hand keypoint position data of the multiple captured images; and perform a handwriting input operation based on the multiple finger trajectory coordinates through a third-party application, wherein the processor also executes the plurality of instructions to: provide the image sequence captured by the camera device to a middleware module through a Device Media Foundation Translator (DMFT) module; and transfer the image sequence to a gesture analysis module in the application layer through the middleware module, wherein the gesture analysis module inputs each of the multiple captured images to the hand keypoint detection model.

Images