Pen tip tracking methods, media, and computing devices

Abstract

The application discloses a pen tip tracking method, medium and computing device, wherein the method comprises: acquiring a pen operation video; acquiring a template image from the pen operation video by using a specific detection model, the template image comprising a pen tip to be tracked, the specific detection model being trained based on a training sample set comprising a plurality of different pen tip images; and determining a pen tip position in the pen operation video based on the template image, the pen operation video and a tracking model constructed based on a twin network. The pen tip tracking method can not only accurately identify pen tip images in pen operation videos in different environments, but also effectively improve the ability of the tracking task to resist the influence of shooting conditions such as light and shadow, while ensuring the robustness, real-time performance and accuracy of the tracking task.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image tracking, in particular to a pen tip tracking method, medium and computing device. BACKGROUND

[0002] In the current pen tip tracking task, the pen tip is mostly regarded as a small target, and then a template matching method is used to realize tracking. However, the pen tip target is different from the ordinary small target, and is easily affected by light, shadow and other factors in the writing process, resulting in a decrease in tracking accuracy, and even a miss phenomenon when the writing speed is too fast. SUMMARY

[0003] The main purpose of the present application is to provide a pen tip tracking method, medium and computing device, which aims to solve the problems mentioned in the background art.

[0004] To achieve the above purpose, the present application provides a pen tip tracking method, comprising:

[0005] Obtaining a pen operation video;

[0006] Using a specific detection model to obtain a template image from the pen operation video, the template image comprising a pen tip to be tracked, the specific detection model being trained based on a training sample set comprising a plurality of different pen tip images;

[0007] Based on the template image, the pen operation video and a tracking model constructed based on a twin network, determining the position of the pen tip in the pen operation video.

[0008] Optionally, the specific detection model is trained based on a training sample set comprising a plurality of different pen tip images, comprising:

[0009] Obtaining a plurality of pen tip tracking video segments;

[0010] Frame-by-frame splitting the plurality of pen tip tracking video segments to obtain a plurality of frame image data;

[0011] Based on the plurality of frame image data, obtaining a plurality of pen tip images in all the frame image data, and constructing a training sample set based on the plurality of pen tip images;

[0012] The specific detection model is trained based on the training sample set, so that the specific detection model can automatically detect pen tip images.

[0013] Optionally, after the training sample set is constructed, the training sample set is normalized, and the specific detection model is trained based on the normalized training sample set.

[0014] Optionally, the plurality of pen tip tracking video clips comprise: a plurality of pen tip tracking video clips taken under same shooting conditions and different shooting conditions.

[0015] Optionally, the plurality of pen tip tracking video clips taken under same shooting conditions comprise:

[0016] a plurality of pen tip tracking video clips taken under same shooting angle, shooting light, and shooting background.

[0017] Optionally, the plurality of pen tip tracking video clips taken under different shooting conditions comprise:

[0018] a plurality of pen tip tracking video clips taken under different shooting angle, shooting light, and shooting background.

[0019] Optionally, the obtaining of the plurality of pen tip images from the plurality of frame image data comprises:

[0020] detecting and obtaining the plurality of pen tip images from the plurality of frame image data by using a specific tool.

[0021] Optionally, the plurality of pen tip tracking video clips comprise hard pen tip tracking video clips and soft pen tip tracking video clips.

[0022] Optionally, the obtaining of the template image from the pen writing video by using a specific detection model comprises:

[0023] taking the first pen tip image detected from the pen writing video by the specific detection model as the template image.

[0024] Optionally, the determining of the pen tip position in the pen writing video based on the template image, the pen writing video, and a tracking model constructed based on a twin network comprises:

[0025] inputting the first pen tip image into a twin network of the tracking model to obtain a first feature response of the first pen tip image;

[0026] taking the first feature response as a target type, tracking a position of the target type on a frame image to which the first pen tip image belongs, and taking the position as a pen tip position in the frame image to which the first pen tip image belongs.

[0027] Optionally, the determining of the pen tip position in the pen writing video based on the template image, the pen writing video, and a tracking model constructed based on a twin network further comprises:

[0028] inputting the first pen tip image into a twin network of the tracking model to obtain a first feature response of the first pen tip image;

[0029] inputting each frame image after the frame where the first nib image is located in the writing video into a twin network of the tracking model to obtain a respective second feature response of each frame image after the frame where the first nib image is located in the writing video;

[0030] matching the second feature response with the first feature response as a target type, and taking the position of the target type matched on the second feature response as the nib position in the writing video.

[0031] Optionally, matching the second feature response with the first feature response as a target type, and taking the position of the target type matched on the second feature response as the nib position in the writing video comprises:

[0032] performing convolution cross-correlation calculation on the first feature response and the second feature response to obtain a response distribution result of each frame image after the frame where the first nib image belongs in the writing video;

[0033] mapping the response distribution result to each frame image corresponding to the writing video, and selecting the position with the highest response score as the nib position in each frame image.

[0034] Optionally, after obtaining the writing video, the method further comprises frame splitting the writing video to obtain a plurality of writing video frame data, and performing normalization processing on the writing video frame data, the specific detection model obtains the template image based on the normalized writing video frame data, and determines the nib position.

[0035] The application further provides a medium having a computer program stored thereon, the computer program being executed by a processor to implement the method of any one of the above.

[0036] The application further provides a computing device comprising a processor configured to implement the method of any one of the above when executing a computer program stored in a memory.

[0037] The technical solution of the application first trains a detection model based on a training sample set comprising a plurality of different nib images, so that the detection model has strong migration ability and can adapt to a wider application environment, and thus can automatically identify various different nib images when tracking the nib; secondly, after automatically identifying the nib image, a tracking model constructed based on a twin network is used to track the identified nib image, thereby ensuring the robustness, real-time performance and accuracy of the tracking task. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and for those skilled in the field, other drawings can be obtained from the technical solutions shown in the drawings without any creative effort.

[0039] Figure 1 A flow chart for the pen tip tracking method according to an embodiment of the present application;

[0040] Figure 2 A flow chart for the pen tip tracking method according to an embodiment of the present application;

[0041] Figure 3 A structure diagram of the detection model in the pen tip tracking method according to an embodiment of the present application;

[0042] Figure 4 A structure diagram of the tracking model in the pen tip tracking method according to an embodiment of the present application;

[0043] Figure 5 A structure diagram of the tracing system in the pen tip tracking method according to an embodiment of the present application;

[0044] Figure 6 A structure diagram of a medium according to an embodiment of the present application;

[0045] Figure 7 A structure diagram of a computing device according to an embodiment of the present application.

[0046] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0047] The principles and spirits of the present application will be described below with reference to several exemplary embodiments. It should be appreciated that these embodiments are presented only to enable those skilled in the art to better understand and implement the present application, and do not limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0048] Those skilled in the art will appreciate that the embodiments of the present application can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure can be embodied in the form of an entirely hardware, an entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0049] According to the embodiments of the present application, a pen tip tracking method, a medium and a computing device are provided. SUMMARY

[0051] The inventors have found that some of the current pen tip tracking methods incorporate Kalman filtering on the basis of matching algorithms, but such methods are susceptible to complex background interference, resulting in target loss and tracking errors. Some other methods use improved particle filtering methods to track the pen tip, which improves the loss situation, but is still susceptible to light, shadow and other factors. Some other methods use a combination of template matching and pen tip shape judgment to achieve the tracking task, which is largely limited by the shape of the pen tip and cannot adapt to the application scenario of soft brush calligraphy.

[0052] The inventors have also found that deep learning technology has excellent performance in image processing and video processing fields, and has shown very good performance in the widely used recognition and detection fields. For example, the local perception characteristics of deep neural networks help small target tracking tasks. Local perception proposes that each neuron does not need to perceive all the pixels in the image, but only needs to perceive the local pixels of the image. Different layers of neural units use local connection, i.e. each layer of neural units is connected to only part of the neural units of the previous layer. Such local connection mode ensures that the learned model parameters have the strongest response to spatial local patterns. This network structure has high invariance to translation, scaling, tilting or other forms of deformation. Moreover, the matching ability based on the twin network structure to achieve the tracking task has become a hot topic in the field of machine learning in recent years. Therefore, introducing the twin network structure into the small target pen tip tracking task can effectively improve the ability of the pen tip tracking task to resist the influence of light, shadow and other factors, while ensuring the robustness, real-time performance and accuracy of the tracking task. In addition, the inventors have found that in the existing target tracking methods based on twin networks, the tracking target needs to be manually selected before tracking, which is not suitable for pen tip tracking scenarios. Therefore, according to the above characteristics, the present application proposes a pen tip tracking method, which pre-trains a detection model, and then uses the trained detection model to detect the pen tip tracking video without human intervention, not only achieving automatic detection and tracking, but also overcoming the problem of being susceptible to light, shadow and other factors in the prior art. After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail.

[0053] Exemplary method

[0054] The pen tip tracking method according to the exemplary embodiments of the present application will be described below with reference to Figure 1 including the following steps:

[0055] Step S100: Obtain a pen video.

[0056] Step S200: obtaining a template image from the writing video by using a specific detection model, the template image including a pen nib to be tracked, the specific detection model being trained based on a training sample set including a plurality of different pen nib images.

[0057] Step S300: determining the pen nib position in the writing video based on the template image, the writing video, and a tracking model constructed based on a twin network.

[0058] For step S200, a template image is obtained from the writing video by using a specific detection model, the template image including a pen nib to be tracked, the specific detection model being trained based on a training sample set including a plurality of different pen nib images. First, the specific detection model needs to be obtained, including the following steps:

[0059] Step S210: obtaining a plurality of pen nib tracking videos; in this step, the plurality of pen nib tracking videos include a plurality of pen nib tracking video segments under the same shooting condition and different shooting conditions, such as: a same video including a plurality of pen nib tracking video segments under the same shooting condition and different shooting conditions, or all segments in a same pen nib tracking video using the same shooting condition, and the shooting conditions of the plurality of pen nib tracking videos being different from each other.

[0060] In addition, the plurality of pen nib tracking videos can be prepared in advance and provided according to a preset interface or uploading manner, for example, can be prepared in advance for a pen nib to be tracked by a user. Or can be a pen nib tracking video shot on site, such as: using a pen to be tracked by a user, writing on site and shooting. Whether prepared in advance or shot on site, as long as it includes pen nib tracking video segments under the same shooting condition and pen nib tracking video segments under different shooting conditions, for example, including: a plurality of pen nib tracking video segments shot under the same shooting light, same shooting angle, and same shooting background, and a plurality of pen nib tracking video segments shot under different shooting angles, different shooting lights, and different shooting backgrounds. Or, a plurality of pen nib tracking video segments shot under the same shooting light, different shooting angles, and same shooting background.

[0061] In summary, when preparing pen nib tracking video data, either a pre-prepared one can be used and provided according to a preset interface or uploading manner, or a real-time shot one can be used, as long as it includes pen nib tracking video segments under the same shooting condition and pen nib tracking video segments under different shooting conditions.

[0062] Step S220: frame-by-frame splitting of the plurality of pen tip tracking videos to obtain a plurality of frame image data; in this step, a frame splitting plug-in can be used, or a frame splitting plug-in can be written, to perform frame-by-frame splitting on all the pen tip tracking videos obtained in step S210, so as to obtain each frame image of each segment of pen tip tracking video, thereby forming a plurality of frame image data.

[0063] Step S230: based on the plurality of frame image data, obtaining a plurality of pen tip images in all the frame image data, and constructing a training sample set based on the plurality of pen tip images; in this step, based on the plurality of frame image data obtained in step S230, a frame image data set {X i} can be established, wherein each frame image can be represented as X ij , which represents the jth frame image in the ith video. Then, for each frame image in the frame image data set, the pen tip image is detected and obtained, such as manually framing the relatively small pen tip image in each frame image in the frame image data set {X i} using a labelme tool, a labelimg tool, and a yolo-mark tool; or automatically framing the relatively small pen tip image in each frame image in the frame image data set {X i} using a Vatic tool, a Sloth tool, and a Rectlabel tool.

[0064] In addition, for the pen tip tracking videos obtained in step S210, there can be individual frames without pen tip images, at this time, the frame image can be directly ignored, and only the pen tip part in the frame image with the pen tip image needs to be framed, so that all the pen tip images framed constitute the training sample set. The training sample set can be represented as {Z}, and Z i represents the ith pen tip template image in the training sample set. It should be noted that when framing the pen tip image, the above-mentioned tools such as labelme can be used, and in other embodiments, other framing tools other than the above-mentioned framing tools can also be used, and the technical solution of the present application does not limit the framing tool.

[0065] Then, step S240: training based on the training sample set to obtain a specific detection model. In this step, an existing detection model can be used, or a detection model can be constructed for training. For example, Figure 3The structure diagram of the two-dimensional convolution network detection model constructed by the embodiment is shown, and the detection model comprises a first two-dimensional convolution layer conv2D_1, a first residual module residual_block_1, a second residual module residual_block_2, a third residual module residual_block_3, a fourth residual module residual_block_4, a fifth residual module residual_block_5 connected in sequence, wherein the third residual module is further connected with a first concatenation layer concatendate_1, the first concatenation layer concatendate_1 is connected with a first two-dimensional convolution block conv2D_block_1, and the first two-dimensional convolution block conv2D_block_1 is further connected with a second two-dimensional convolution layer conv2D_2; in addition, the fourth residual module residual_block_4 is further connected with a second concatenation layer concatendate_2, the second concatenation layer concatendate_2 is connected with a second two-dimensional convolution block conv2D_block_2, the second two-dimensional convolution block conv2D_block_2 is connected with a first upsampling module upsampling2D_1, the first upsampling module upsampling2D_1 is connected with a third two-dimensional convolution layer conv2D_3, the third two-dimensional convolution layer conv2D_3 is connected with the first concatenation layer concatendate_1, and the second two-dimensional convolution block conv2D_block_2 is further connected with a fourth two-dimensional convolution layer conv2D_4; in addition, the fifth residual module residual_block_5 is connected with a fifth two-dimensional convolution layer conv2D_5 and a sixth two-dimensional convolution layer conv2D_6 respectively, the fifth two-dimensional convolution layer conv2D_5 is further connected with a second upsampling module upsampling_2, and the second upsampling module upsampling_2 is connected with the second concatenation layer concatendate_2.The first two-dimensional convolution layer conv2D_1 has a convolution kernel of 3x3 and a channel number of 32; the second two-dimensional convolution layer conv2D_2 has a convolution kernel of 1x1 and a channel number of 256; the third two-dimensional convolution layer conv2D_3 has a convolution kernel of 1x1 and a channel number of 128; the fourth two-dimensional convolution layer conv2D_4 has a convolution kernel of 1x1 and a channel number of 128; the fifth two-dimensional convolution layer conv2D_5 has a convolution kernel of 1x1 and a channel number of 256; the sixth two-dimensional convolution layer conv2D_5 has a convolution kernel of 11x11 and a channel number of 256; the first residual module residual_block_1 has a convolution kernel of 1x1 and a channel number of 64; the second residual module residual_block_2 has a convolution kernel of 2x2 and a channel number of 128; the third residual module residual_block_3 has a convolution kernel of 8x8 and a channel number of 256; the fourth residual module residual_block_4 has a convolution kernel of 8x8 and a channel number of 512; the fifth residual module residual_block_1 has a convolution kernel of 4x4 and a channel number of 1024; the first two-dimensional convolution block conv2D_block_1 and the second two-dimensional convolution block conv2D_block_2 each include three groups of convolution layers connected in sequence, wherein each group of convolution layers includes a convolution layer with a channel number of 128 and a convolution kernel of 1x1, and a convolution layer with a channel number of 256 and a convolution kernel of 3x3. The detection model structure is relatively light and fast in processing speed, so as to be able to detect the pen video in real time. After the detection model is constructed, the pen tip image in the training sample set established in step S230 can be used to train the detection model. As known from step S210, the pen tip images in the training sample set include pen tip images under various shooting conditions, such as pen tip images under the same shooting light, the same shooting angle, the same shooting background, and different shooting light, different shooting angle, and different shooting background; or pen tip images with some same shooting conditions and some different shooting conditions, such as the same shooting light and different shooting angle, the same shooting angle and different shooting background, etc. Thus, the training sample set contains various forms of pen tip images under different combinations of various light, various background, various angle, various shadow, etc. Therefore, after the pen tip images in the training sample set are used to train the detection model, the detection model has strong migration ability and can adapt to a wider application environment, so that the pen tip image can be automatically detected and recognized from the pen video under various shooting conditions.

[0066] After the specific detection model is constructed, a template image is obtained from the penmanship video using this specific detection model. The penmanship video can be input into the trained specific detection model, and the first pen tip image detected by the specific detection model from the penmanship video can be used as the template image. Since not every frame of the penmanship video may contain a pen tip image, it is only necessary to input the penmanship video into the specific detection model in real time. When the specific detection model detects the first frame in the penmanship video that contains a pen tip image, it can automatically recognize this first pen tip image.

[0067] This concludes the complete explanation of the construction steps for a specific detection model, and how the template image in the penmanship video was obtained based on the constructed specific detection model.

[0068] Next, step S300 is performed, determining the pen tip position in the pen movement video based on the template image, the pen movement video, and the tracking model constructed based on a Siamese network. For example... Figure 4 As shown, the tracking model comprises a first convolutional layer (conv_1), a first pooling layer (pool_1), a second convolutional layer (conv_2), a second pooling layer (pool_2), a third convolutional layer (conv_3), a fourth convolutional layer (conv_4), and a fifth convolutional layer (conv_5), connected sequentially. The specific parameters are as follows: conv_1 has an 11×11 kernel size, a stride of 2, and 96 channels; pool_1 has a 3×3 kernel size and a stride of 2; conv_2 has a 5×5 kernel size, a stride of 1, and 256 channels; pool_2 has a 3×3 kernel size and a stride of 2; conv_3 has a 3×3 kernel size, a stride of 1, and 384 channels; conv_4 has a 3×3 kernel size, a stride of 1, and 384 channels; and conv_5 has a 3×3 kernel size, a stride of 1, and 256 channels. This tracking model's network structure offers the advantage of being more lightweight and allows for faster data processing. On the other hand, after step S200 identifies the first pen tip image, for ease of description, the frame of the pen movement video containing the first pen tip image is referred to as the first frame (the first frame here is only for ease of description and does not necessarily represent the first frame image of the pen movement video). Therefore, the pen tip begins to appear in the first frame, and the pen tip position in the pen movement video includes the pen tip position in the first frame, as well as the pen tip position in every frame after the first frame. The method for determining the pen tip position in the first frame is as follows:

[0069] input the first pen tip image into the twin network of the tracking model to obtain a first feature response of the first pen tip image; then take the first feature response as a target type to track the position of the target type on a frame image in the writing video to which the first pen tip image belongs (i.e. track the position of the target type on the first frame), and take the position of the first feature response on the first frame as the pen tip position in the frame image in the writing video to which the first pen tip image belongs (i.e. take the position of the first feature response on the first frame as the pen tip position in the first frame).

[0070] The pen tip position determination method for each frame after the first frame is as follows:

[0071] input the first pen tip image into the twin network of the tracking model to obtain a first feature response of the first pen tip image;

[0072] input each frame image in the writing video after the frame in which the first pen tip image is located into the twin network of the tracking model (i.e. input each frame after the second frame into the twin network), to obtain a second feature response of each frame image in the writing video after the frame in which the first pen tip image is located (i.e. the second feature response corresponding to the second frame and each frame after the second frame);

[0073] match the second feature response with the first feature response as a target type, and take the position of the target type matched on the second feature response as the pen tip position in the writing video.

[0074] In combination with Figure 2 The pen tip position determination method is described as follows:

[0075] First, the writing video is real-time frame-by-frame split, and then the split frame images are real-time transmitted to a specific detection model. Assuming that the pen tip image appears in the T0 frame of the writing video, the specific detection model automatically recognizes the pen tip image in the T0 frame image, takes the pen tip image as a template image, inputs the pen tip image into the twin network, extracts a first feature response, takes the first feature response as a target type to track the position thereof on the T0 frame image, and takes the position as the pen tip position on the T0 frame image and outputs; on the other hand, each frame image after the T0 frame of the writing video is output to the twin network, and a second feature response of each frame is extracted, then the first feature response is taken as a target type, and the target type of the first feature response is matched on the second feature response of each frame image after the T0 frame, and the matched target position (which can be expressed by coordinates) is taken as the pen tip position on each frame after the T0 frame and output.

[0076] The matching can be performed by the following method:

[0077] f(T0) represents the first feature response extracted from the pen tip image detected by the twin network from the T0th frame image;

[0078] f(T) represents the second feature response extracted from the Tth frame image by the twin network, where T is after T0;

[0079] The second feature response and the first feature response extracted from the Tth frame image are convolved with each other to obtain f(T0, T) = f(T0) * f(T), where "*" represents the convolution cross-correlation operation, and f(T0, T) is the response distribution result of the pen tip image in the T0th frame image of the pen video in the Tth frame image of the pen video;

[0080] The response distribution result f(T0, T) obtained is then mapped back to the Tth frame image, and the region with the highest response score is selected as the position result of the tracking target of the Tth frame image;

[0081] When the position of each frame image of the pen video starting from the T0th frame image is determined, the entire pen tip tracking result of the pen video is obtained.

[0082] In another embodiment of the present embodiment, after the pen tip image detection is performed on each frame image based on the frame image data and the training sample set is established, the method further includes: performing normalization processing on the pen tip images in the training sample set, and training the detection model using the pen tip images in the training sample set after the normalization processing. Specifically, the maximum or minimum normalization processing method can be used to process the pen tip images in the training sample set, and the same form of standard pen tip image is obtained after the normalization processing. Then, it is easier and more convenient to train the detection model using the same form of standard pen tip image.

[0083] In another embodiment of the present embodiment, the plurality of pen tip tracking videos includes hard pen tip tracking videos and soft pen tip tracking videos. The hard pen tip tracking videos and the soft pen tip tracking videos are only different in pen tip type, and the acquisition method is the same as that in step S210, which is not described here. Therefore, since the hard pen tip and the soft pen tip are covered, the soft pen tip images and the hard pen tip images under various conditions are included in the training sample set, and therefore the detection model can not only automatically recognize the hard pen tip image, but also automatically recognize the soft pen tip image after being trained using the pen tip images in the training sample set.

[0084] In another embodiment of the present embodiment, the plurality of pen tip tracking videos includes a hard pen tip tracking video and a soft pen tip tracking video, wherein the hard pen tip tracking video includes pen tip tracking video segments of the same shooting condition and pen tip tracking video segments of different shooting conditions, and the soft pen tip tracking video also includes pen tip tracking video segments of the same shooting condition and pen tip tracking video segments of different shooting conditions.

[0085] In another embodiment of the present embodiment, after the writing video is obtained, the writing video is frame-split to obtain a plurality of writing video frame data, and the writing video frame data is normalized, the specific detection model obtains a template image based on the normalized writing video frame data, and the pen tip position is determined. For example, after the writing video is obtained, the writing video is frame-split by using a frame-split tool to obtain each frame image, then based on all the frame images, a maximum value or minimum value normalization processing method is used for normalization processing, and then each frame image of the writing video with the same form is obtained, and then the input is detected to the detection model to detect the pen tip image, which is more easy and fast.

[0086] From the method of the above exemplary embodiment, it can be known that the technical solution of the present application first obtains pen tip tracking video data under different shooting conditions, then frame-by-frame cuts the pen tip tracking video data under various shooting conditions, and obtains the pen tip image in each frame to establish a training sample set, and then uses the training sample set to train a detection model. Since the training sample set contains pen tip tracking videos under various shooting conditions, the training sample set obtained also contains pen tip images under various shooting conditions, so that the trained detection model can accurately identify the pen tip image in the writing video under different environments when detecting the writing video. Then, when the detection model detects the first pen tip image, the detected first pen tip image and the writing video are input into a twin network, and convolution calculation is performed on the first pen tip image and the frame image of the writing video through the twin network, so that the position of the first pen tip image in each frame of the writing video can be obtained, thereby forming a pen tip tracking result. On the other hand, since the twin network is used to perform convolution calculation on the first pen tip image and the frame image of the writing video, the ability of the tracking task to resist the influence of shooting conditions such as light and shadow is effectively improved, and the robustness, real-time performance and accuracy of the tracking task are also ensured.

[0087] Exemplary medium

[0088] After introducing the method and device of the exemplary embodiment of the present application, next, with reference to Figure 6 The computer-readable storage medium of the exemplary embodiment of the present application is described.

[0089] Please refer to Figure 6The computer readable storage medium shown is an optical disc 200, on which a computer program (i.e. program product) is stored, which, when run by a processor, implements each step described in the above method embodiments, for example:

[0090] Obtaining a writing video;

[0091] Obtaining a template image from the writing video by using a specific detection model, the template image including a pen nib to be tracked, the specific detection model being trained based on a training sample set including a plurality of different pen nib images;

[0092] Determining the pen nib position in the writing video based on the template image, the writing video and a tracking model constructed based on a twin network.

[0093] It should be noted that examples of the computer readable storage medium can also include, but are not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory or other optical, magnetic storage medium, which will not be described one by one here.

[0094] Exemplary computing device

[0095] After introducing the method, device and medium of the exemplary embodiments of the application, next, with reference to Figure 7 The computing device 300 of the exemplary embodiments of the application is described, Figure 7 A block diagram of an exemplary computing device 300 suitable for implementing embodiments of the application is shown, which can be a computer system or a server. Figure 7 The computing device 300 shown is merely an example and should not bring any limitation to the function and use range of the embodiments of the application.

[0096] As Figure 7 shown, the components of the computing device 300 can include, but are not limited to, one or more processors or processing units 310, system memory 320, a bus 330 connecting different system components, including system memory and processing unit 310.

[0097] The computing device 300 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device 300, including volatile and non-volatile media, removable and non-removable media.

[0098] The system memory can include computer system readable media in the form of volatile memory, such as random access memory (RAM 321) and / or cache memory 322. The computing device 300 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, ROM 323 can be used for reading a static, non- volatile, magnetic media, e.g., a read only memory (ROM) or a Figure 7 hard disk drive (HDD) typically referred to as a "hard disk drive" or "hard drive" is typically provided in the computing device 300. It should be appreciated that the ROM 323 could also be used to store static, non-volatile, magnetic media (e.g., a hard disk drive), although it is typically not used in this capacity. A removable / non-removable, volatile / non-volatile computer system storage medium can also be used in the computing device 300. It should be appreciated that a Figure 7 floppy disk drive, a magnetic disk drive, and other removable / non-removable, volatile / non-volatile computer system storage media can be used for the same purposes. In this regard, the data storage media can be connected to the system bus by a data media drive interface. The data storage media such as the ROM 323 can also contain the one or more programs, such as the operating system 324 and one or more application programs 325, which can be used in whole or in part to implement embodiments of the present application.

[0099] The programs (including program modules 324) stored on the system memory, such as the ROM 323 and the RAM 321, can include an operating system 324, one or more application programs 325, other program modules 326, and program data 327. Each of the operating system 324, one or more application programs 325, other program modules 326, and program data 327 (or some combination thereof), can implement aspects of the application and can utilize data storage 328, for example, as described herein. It should be appreciated that the application can be implemented with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like.

[0100] The computing device 300 can also communicate with one or more external devices 340 such as a keyboard, a pointing device, a display, etc. via an input / output (I / O) interface 350. Further, the computing device 300 can communicate with one or more networks such as a local area network (LAN), a wide area network (WAN), and / or the public network such as the Internet, via a network adapter 360. As Figure 7 illustrated, the network adapter 360 can be communicatively coupled to the other components of the computing device 300, such as the processing unit 310, via a bus. It should be appreciated that the network adapter 360 can also be utilized to communicate with other electronic devices and systems which are not illustrated, such as a network of computing devices. Figure 7 It should be appreciated that the application can be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like.

[0101] The processing unit 310 can execute various functions and data processing by running programs stored in the system memory, such as:

[0102] acquiring a pen video;

[0103] acquire a template image from the writing video, the template image including a pen nib to be tracked, the specific detection model being trained based on a training sample set including a plurality of different pen nib images;

[0104] determine the pen nib position in the writing video based on the template image, the writing video, and a tracking model constructed based on a twin network.

[0105] An exemplary tracing system

[0106] Reference Figure 5 An exemplary tracing system of the present application is described, Figure 5 A structure diagram of an exemplary tracing system suitable for implementing an embodiment of the present application is shown, which includes a tracing board 400, a camera 420, a computer 430, and a display screen 440, wherein the camera 420 is used to capture a writing video of a tracing pen 410 on the tracing board 400, the camera 420 is connected with the computer 430 and sends the captured writing video to the computer 430, the computer 430 can be specifically referred to the exemplary computing device described above, the computer 430 is used to execute the pen nib tracking method described above to determine the movement position of the pen nib according to the writing video captured by the camera 420, and the computer 430 is also connected with the display screen 440 and used to display the determined pen nib position on the display screen 440 for reference of a tracing person.

[0107] In addition, although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps.

[0108] Although the spirit and principles of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the disclosed specific embodiments, and the division of aspects does not mean that the features in these aspects cannot be combined for the benefit, but only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

[0109] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made by using the content of the present application specification and drawings, or directly / indirectly applied in other related technical fields is included in the patent protection scope of the present application.

[0110] Through the above description, the embodiments of the present application provide the following technical solutions, but are not limited thereto:

[0111] 1. A pen tip tracking method, comprising:

[0112] obtaining a writing video;

[0113] obtaining a template image from the writing video by using a specific detection model, the template image comprising a pen tip to be tracked, the specific detection model being trained based on a training sample set comprising a plurality of different pen tip images;

[0114] determining a pen tip position in the writing video based on the template image, the writing video, and a tracking model constructed based on a twin network.

[0115] 2. The pen tip tracking method of claim 1, wherein the specific detection model is trained based on a training sample set comprising a plurality of different pen tip images, comprising:

[0116] obtaining a plurality of pen tip tracking video clips;

[0117] frame-by-frame splitting the plurality of pen tip tracking video clips to obtain a plurality of frame image data;

[0118] obtaining a plurality of pen tip images from all of the frame image data based on the plurality of frame image data, and constructing a training sample set based on the plurality of pen tip images;

[0119] the specific detection model being trained based on the training sample set, so that the specific detection model can automatically detect pen tip images.

[0120] 3. The pen tip tracking method of claim 1 or 2, further comprising, after the training sample set is constructed, normalizing the training sample set, the specific detection model being trained based on the normalized training sample set.

[0121] 4. The pen tip tracking method of any one of claims 1-3, wherein the plurality of pen tip tracking video clips comprises a plurality of pen tip tracking video clips shot under same shooting conditions and different shooting conditions.

[0122] 5. The pen tip tracking method of any one of claims 1-4, wherein the plurality of pen tip tracking video clips shot under same shooting conditions comprises:

[0123] a plurality of pen tip tracking video clips shot under same shooting angle, shooting light, and shooting background;

[0124] the plurality of pen tip tracking video clips shot under different shooting conditions comprises:

[0125] The multiple pen tip tracking video clips are captured from different angles, under different lightings, and different backgrounds.

[0126] 6. The pen tip tracking method of any one of claims 1-5, wherein the obtaining, based on the multiple frames of image data, multiple pen tip images from all of the frames of image data comprises:

[0127] The multiple pen tip images are detected from all of the frames of image data based on the multiple frames of image data and using a specific tool.

[0128] 7. The pen tip tracking method of any one of claims 1-6, wherein the multiple pen tip tracking video clips comprise hard pen tip tracking video clips and soft pen tip tracking video clips.

[0129] 8. The pen tip tracking method of any one of claims 1-7, wherein the obtaining, from the writing video, a template image using a specific detection model comprises:

[0130] The first pen tip image detected from the writing video by the specific detection model is used as the template image.

[0131] 9. The pen tip tracking method of any one of claims 1-8, wherein the determining, based on the template image, the writing video, and a tracking model constructed based on a twin network, a pen tip position in the writing video comprises:

[0132] The first pen tip image is input into the twin network of the tracking model to obtain a first feature response of the first pen tip image;

[0133] The position of the target type in the frame of image to which the first pen tip image belongs is tracked based on the first feature response as the target type, and is used as the pen tip position in the frame of image to which the first pen tip image belongs.

[0134] 10. The pen tip tracking method of any one of claims 1-9, wherein the determining, based on the template image, the writing video, and a tracking model constructed based on a twin network, a pen tip position in the writing video further comprises:

[0135] The first pen tip image is input into the twin network of the tracking model to obtain a first feature response of the first pen tip image;

[0136] Each frame of image after the frame in which the first pen tip image is located in the writing video is input into the twin network of the tracking model to obtain a second feature response of each frame of image after the frame in which the first pen tip image is located in the writing video;

[0137] matching the second feature response to the target type of the first feature response, and taking the position of the target type matched on the second feature response as the pen tip position in the pen video.

[0138] 11.The pen tip tracking method of any one of claims 1-10, wherein matching the second feature response to the target type of the first feature response, and taking the position of the target type matched on the second feature response as the pen tip position in the pen video comprises:

[0139] performing a convolution cross-correlation calculation on the first feature response and the second feature response to obtain a response distribution result of the first pen tip image in each frame image after the frame image to which the first pen tip image belongs in the pen video;

[0140] mapping the response distribution result to each frame image corresponding to the pen video, and taking the position with the highest response score as the pen tip position in each frame image.

[0141] 12.The pen tip tracking method of any one of claims 1-11, wherein after obtaining the pen video, the method further comprises frame splitting the pen video to obtain a plurality of pen video frame data, and performing normalization processing on the pen video frame data, the specific detection model obtains the template image based on the normalized pen video frame data, and determines the pen tip position.

[0142] 13.A medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method of any one of claims 1-12.

[0143] 14.A computing device, comprising a processor configured to implement the method of any one of claims 1-12 when executing a computer program stored in a memory.

Claims

1. A pen tip tracking method, comprising: acquiring a writing video; acquiring a template image from the writing video by using a specific detection model, the template image comprising a pen tip to be tracked, the specific detection model being trained based on a training sample set comprising a plurality of different pen tip images; determining a pen tip position in the writing video based on the template image, the writing video, and a tracking model constructed based on a twin network; detecting a first pen tip image from the writing video by using the specific detection model as the template image; training the specific detection model based on a training sample set comprising a plurality of different pen tip images, comprising: acquiring a plurality of pen tip tracking video segments; frame-by-frame splitting the plurality of pen tip tracking video segments to obtain a plurality of frame image data; acquiring a plurality of pen tip images from all the frame image data based on the plurality of frame image data, and constructing a training sample set based on the plurality of pen tip images; the specific detection model being trained based on the training sample set, so that the specific detection model can automatically detect pen tip images; the plurality of pen tip tracking video segments comprising a plurality of pen tip tracking video segments shot under the same shooting condition and different shooting conditions; the specific detection model comprising a first two-dimensional convolution layer, a first residual module, a second residual module, a third residual module, a fourth residual module, and a fifth residual module connected in sequence, wherein the third residual module is further connected with a first concatenation layer, the first concatenation layer is connected with a first two-dimensional convolution block, the first two-dimensional convolution block is further connected with a second two-dimensional convolution layer; the fourth residual module is further connected with a second concatenation layer, the second concatenation layer is connected with a second two-dimensional convolution block, the second two-dimensional convolution block is connected with a first up-sampling module, the first up-sampling module is connected with a third two-dimensional convolution layer, the third two-dimensional convolution layer is connected with the first concatenation layer, the second two-dimensional convolution block is further connected with a fourth two-dimensional convolution layer; the fifth residual module is connected with a fifth two-dimensional convolution layer and a sixth two-dimensional convolution layer, the fifth two-dimensional convolution layer is further connected with a second up-sampling module, and the second up-sampling module is connected with the second concatenation layer.

2. The pen tip tracking method of claim 1, wherein, Further comprising, after the training sample set is constructed, performing normalization processing on the training sample set, and the specific detection model is trained based on the normalized training sample set.

3. The pen tip tracking method of claim 1, wherein, the plurality of pen tip tracking video segments shot under the same shooting condition comprising: a plurality of pen tip tracking video segments shot under the same shooting angle, the same shooting light, and the same shooting background; the plurality of pen tip tracking video segments shot under different shooting conditions comprising: a plurality of pen tip tracking video segments shot under different shooting angles, different shooting lights, and different shooting backgrounds.

4. The pen tip tracking method of any one of claims 1-3, wherein, the acquiring a plurality of pen tip images from all the frame image data based on the plurality of frame image data comprising: detecting a plurality of pen tip images from all the frame image data based on the plurality of frame image data.

5. The pen tip tracking method of any one of claims 1-3, wherein, the plurality of pen tip tracking video segments comprising hard pen tip tracking video segments and soft pen tip tracking video segments.

6. The pen tip tracking method of claim 1, wherein, determining the nib position in the writing video based on the template image, the writing video, and a tracking model constructed based on a twin network, comprises: inputting the first nib image into a twin network of the tracking model to obtain a first feature response of the first nib image; tracking a position of the target type on a frame image to which the first nib image belongs, taking the first feature response as a target type, and taking the position as a nib position in the frame image to which the first nib image belongs.

7. The pen tip tracking method of claim 1, wherein, determining the nib position in the writing video based on the template image, the writing video, and a tracking model constructed based on a twin network, further comprises: inputting the first nib image into a twin network of the tracking model to obtain a first feature response of the first nib image; inputting each frame image after the frame in which the first nib image is located in the writing video into the twin network of the tracking model to obtain a second feature response of each frame image after the frame in which the first nib image is located in the writing video; matching the second feature response taking the first feature response as a target type, and taking a position of the target type matched on the second feature response as the nib position in the writing video.

8. The pen tip tracking method of claim 7, wherein, matching the second feature response taking the first feature response as a target type, and taking a position of the target type matched on the second feature response as the nib position in the writing video comprises: performing convolution cross-correlation calculation on the first feature response and the second feature response to obtain a response distribution result of each frame image after the frame image to which the first nib image belongs in the writing video; mapping the response distribution result to each frame image corresponding to the writing video, and selecting a position with the highest response score as the nib position in each frame image.

9. The pen tip tracking method of any one of claims 6-8, wherein, After the writing video is obtained, further comprising frame splitting the writing video to obtain a plurality of writing video frame data, and performing normalization processing on the writing video frame data, and the specific detection model obtains the template image and the nib position determination based on the normalized writing video frame data.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-9.

11. A computing device, comprising: The computing device comprises a processor configured to implement the method of any one of claims 1-9 when the processor executes a computer program stored in a memory. The computing device comprises a processor configured to implement the method of any one of claims 1-9 when the processor executes a computer program stored in a memory.

Images