Writer identification method and system based on spatio-temporal feature fusion and application

Abstract

The invention discloses a writer identification method based on spatio-temporal feature fusion, and the method comprises the steps: extracting spatial representation and time representation from handwriting track data through self-adaption, fusing the representation, and then carrying out the writer identification. Comprising the following steps of 1, collecting real-time writing data of a writer, and constructing an original data set; 2, converting track data in the original data set into a handwritten Chinese character image, and extracting spatial representation of a writer from the handwritten Chinese character image; 3, converting the trajectory data in the original data set into a time sequence feature map, and extracting the time representation of the writer from the time sequence feature map; and step 4, matching and fusing the spatial representation and the time representation obtained by the same writing track data as a final writer identity representation, and discriminating the writer identity based on the representation. The invention further discloses a recognition system for implementing the recognition method, and the recognition system has wide application value.

Description

Technical Field

[0001] The present invention belongs to the field related to deep learning and computer vision, and specifically relates to a method, system and application for effectively identifying the identity of the Chinese character writer by integrating the dynamic time representation in the writing process and the static spatial representation in the writing result. Background Art

[0002] Writer identification refers to the research problem of determining the writer's identity by analyzing and processing handwriting samples and extracting writing style features. According to the input data type, writer identification can be divided into static writer identification [1] (The input data is usually scanned handwritten documents, signature images, and other handwritten text images) and dynamic writer recognition [2] (The input data is usually time series data containing dynamic information during the writing process); according to the content of the recognized text, writer recognition can be divided into text-related writer recognition [3] (the recognition process depends on the specific text content, that is, the writer writes the same text during training and testing) and text-independent writer recognition [4] (The recognition process does not depend on the specific text content. The text content written by the writer during the training and testing phases is different. The recognition model must rely on the writing style characteristics rather than the specific text content for recognition.) Writer recognition technology has a wide range of applications in many fields. For example, it is used in the financial field to prevent forgery and fraud to ensure the security and authenticity of transactions. [5] Verify the identity of candidates in the education field to prevent cheating, and provide support for the development of Chinese character writing evaluation systems [6] .

[0003] The evolution of writer recognition technology has undergone significant development from early traditional methods to modern deep learning methods. In the early days, writer recognition mainly relied on experienced experts to manually design and extract writing features for identification, which was time-consuming, labor-intensive, and extremely inefficient. [7] After the popularization of computers, researchers relied on computers to extract features and applied various statistical and machine learning methods to feature classification. With the expansion of data scale and the increase of computing power, deep learning technology emerged and was used in writer recognition tasks. By automatically extracting writing features, it significantly improved recognition accuracy and efficiency. [8] .

[0004] Although current writer recognition technology has made significant progress, some difficulties and challenges still exist:

[0005] 1) The sparsity of handwritten pixels hinders feature representation: During handwriting, the actual number of handwritten pixels accounts for a relatively small proportion of the total number of pixels in the entire image area. Consequently, effective information reflecting the writer is even more sparse in the image, which is a major factor contributing to the model's low prediction accuracy. While existing recognition model architectures can alleviate this problem to some extent, significant research remains in the design of representation networks.

[0006] 2) Difficulty in adaptively capturing and fusing multi-granular features: Coarse-grained and fine-grained features refer to different levels of feature information extracted from handwritten character images. Fine-grained features refer to pixel-level stroke details extracted from handwritten character images, including pen tip pressure, stroke tilt angle, stroke connection method, and stroke starting and ending points. Coarse-grained features refer to higher-level information extracted from the entire character image, describing the global structure and morphology of the entire character, such as its symmetry, uniformity, tilt, proportion, and overall layout. Existing recognition models have difficulty adaptively capturing and fusing these different levels of features.

[0007] 3) Difficulty in effectively integrating dynamic temporal and static spatial features: Dynamic temporal features describe the transformation trends of the entire handwriting process, while static spatial features describe the structure and details of the handwritten result. These two features often have complex dependencies. Existing recognition methods often focus on features in one dimension, lacking understanding and exploration of the other. Summary of the Invention

[0008] In order to solve the difficulties and challenges of existing writer identification models, the present invention provides a writer identification method based on spatiotemporal feature fusion. The method of the present invention converts the writing trajectory time series data collected by intelligent hardware such as smart dot pens into handwritten Chinese character images and time series feature maps, extracts the spatial representation and temporal representation of the corresponding writer's identity from them, and fuses them as the final writer representation as the basis for discrimination. Writer identification is performed through representations in two dimensions, which has a higher recognition accuracy than previous recognition models and has stronger robustness. Robustness refers to the decrease in the degree of decrease in recognition accuracy caused by the reduction in data scale after the introduction of a special attention mechanism. The writer identification method of the present invention can still maintain a high recognition accuracy when the data scale is limited.

[0009] The workflow of the writer identification method of the present invention comprises the following steps:

[0010] Step 1: Collect the writer's real-time writing data and construct the original data set;

[0011] Step 2: Convert the trajectory data in the original dataset into handwritten Chinese character images, and extract the writer's spatial representation from the handwritten Chinese character images;

[0012] Step 3: Convert the trajectory data in the original dataset into a time series feature graph, and extract the writer's time representation from the time series feature graph;

[0013] Step 4: Match the spatial representation and the temporal representation obtained from the same writing trajectory data, and fuse them together using a weighted fusion strategy to obtain the final writer identity representation, and determine the writer's identity based on the representation.

[0014] In step 1, the writing data is collected through intelligent hardware devices including intelligent dot pen, etc. The writing data includes spatial feature data such as strokes and structure of the Chinese characters themselves and dynamic time series data during the writing process.

[0015] In step 2, the step of converting the trajectory data into a two-dimensional handwritten Chinese character image includes: segmenting all sampling data points according to strokes, extracting valid information of the sampling data points in each segment, connecting the spatial coordinates of all adjacent sampling data points in the same stroke with line segments, determining the thickness of the connecting line segments, and completing the drawing of the handwritten Chinese character image;

[0016] The specific steps are:

[0017] Step i. Sampling point segmentation: segment all sampling data points according to strokes;

[0018] Step ii. Information extraction: extract valid information from the segmented sampling data points and eliminate invalid information;

[0019] The valid information includes spatial position coordinates, pen tip force information, timestamp information, etc.

[0020] Step iii. Stroke connection: Connect the spatial coordinates of all adjacent sampling data points within the same stroke with a line segment and determine the thickness of the connecting line. After connecting all strokes, the handwritten Chinese character image is drawn; the thickness of the connecting line is determined according to the pen tip strength of two adjacent sampling points, and further, determined according to the average pen tip strength of two adjacent sampling points;

[0021] Step iv. Save and name: Save the drawn handwritten Chinese character image, and the name includes the image primary key and writer identification; the image primary key includes the copybook row primary key, copybook row space number, and the Chinese character corresponding to the copybook row.

[0022] In step 2, the handwritten Chinese character image is preprocessed including image cleaning, image scaling, image filling, etc. before extracting spatial features;

[0023] The image cleaning includes removing duplicate and blurred images from the handwritten Chinese character image; the image scaling includes scaling different handwritten Chinese character images to the same size; the image filling includes filling the scaled image in a blank image of target size; and / or,

[0024] Extracting spatial representations from Chinese character images using convolutional neural networks; and / or,

[0025] The convolutional neural network is a GR-RNN network; and / or,

[0026] Adding an attention mechanism after the convolutional block of the fourth convolutional layer in the GR-RNN network to form a spatial representation extraction network to extract the spatial representation of the writer corresponding to the handwritten Chinese character image; and / or,

[0027] The spatial representation extraction network extracts features through multi-granularity sparse convolution, fuses feature maps of different granularities, calculates weights and performs weighted summation to obtain the writer's spatial representation.

[0028] The workflow of the attention mechanism in the spatial representation extraction network includes the following steps:

[0029] Step a. Multi-granularity feature extraction: n sparse convolution kernels of different sizes are applied to the output X of the convolution layer of the recognition network to obtain n feature maps of the same size;

[0030] Step b. Multi-granularity feature fusion: concatenate the n feature maps according to the channel, and then use a convolution kernel of size 1 to compress the channel to its original size to obtain a multi-granularity feature fusion feature map F;

[0031] Effectively fuse multi-granularity spatial features through 1×1 convolution fusion instead of traditional feature map addition;

[0032] Step c. Weight calculation: After the fused feature map passes through the global average pooling layer, multiple fully connected layers and the Softmax activation function layer, the weights of each channel of the spatial features of different granularities are obtained;

[0033] Step d. Weighted summation: perform weighted summation on the weight and the corresponding spatial feature to obtain the final writer spatial representation.

[0034] In step 3, the original trajectory data is converted into a two-dimensional time series feature map using methods such as the relative position matrix method:

[0035] Perform sequence normalization, sequence dimensionality reduction, relative position calculation, and normalization operations on the original trajectory data in a time series to generate three time series feature maps of different dimensions, and splice them into a final three-channel two-dimensional time series feature map according to the channel dimension; and / or,

[0036] The writer's temporal representation is output through the temporal representation extraction network GR-RNN model.

[0037] In step 4, the static writing result and the dynamic writing process are combined by fusing the spatial representation and the temporal representation.

[0038] The present invention integrates the spatial and temporal representations of the writer's identity, thereby enhancing the robustness of the model and the accuracy of the recognition results. When extracting spatial features of different granularities, the sparse convolution kernel structure is used to extract features, which has better stability and performance than traditional convolution kernels, and alleviates the impact of the sparseness of effective information in handwritten Chinese character images. It can adaptively capture and fuse multi-granularity spatial features, and make comprehensive judgments based on the writing process and writing results.

[0039] The present invention also provides an identification system for implementing the above identification method, the identification system comprising: a data acquisition module, a data conversion module, a feature extraction module, a feature fusion module, and an identification module; the data acquisition module, the data conversion module, the feature extraction module, the feature fusion module, and the identification module are connected in sequence;

[0040] The data acquisition module is used to collect the real-time writing data of the writer;

[0041] The data conversion module is used to convert the collected trajectory data into handwritten Chinese character images and time series feature maps;

[0042] The feature extraction module is used to extract spatial features and temporal features from the image and the time series feature map respectively;

[0043] The feature fusion module is used to fuse spatial features and temporal features to generate a writer's identity representation;

[0044] The recognition module is used to determine the identity of the writer based on the fused representation.

[0045] The present invention also provides applications of the above-mentioned recognition method or system in fields including contract authenticity verification, exam cheating prevention, and handwritten document classification.

[0046] The beneficial effects provided by the present invention include: the recognition method of the present invention introduces a specific attention mechanism based on the spatial representation extraction network. The attention mechanism introduces sparse convolution kernels of different sizes to capture spatial features of different granularities, which is equivalent to playing a downsampling role, helping the attention mechanism to focus on important feature areas instead of performing full convolution on the entire input, alleviating the impact of the sparsity of effective information in handwritten Chinese character images. At the same time, the attention mechanism fuses spatial features of different granularities using 1×1 convolution and then calculates weights, assigning different weights to multi-granularity features and performing weighted fusion, taking into account the impact of spatial features of different granularities on the final recognition results. The recognition method of the present invention integrates static spatial features and dynamic temporal features, selects a suitable temporal feature graph conversion algorithm, converts the original trajectory temporal feature graph into a two-dimensional temporal feature graph, uses a temporal representation extraction network to mine the writer's identity temporal representation, and fuses it with the original spatial representation to form the final writer's identity representation. It distinguishes from features in multiple dimensions, enhances the accuracy of the writer recognition result, and improves the robustness and generalization of the recognition network. Through the recognition method of the present invention, the Top 1 recognition accuracy rate on the online handwritten Chinese character dataset reached 91.31%, and the Top 5 recognition accuracy rate reached 98.42%, both reaching a relatively high level. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 This is a partial handwritten Chinese character image converted from the original trajectory data.

[0049] Figure 2 This is the structural diagram of the GR-RNN network.

[0050] Figure 3 Figure 2 is a diagram of the attention mechanism structure used to extract spatial representations.

[0051] Figure 4 Flowchart for converting raw trajectory data into time series feature graph.

[0052] Figure 5 This is a workflow diagram of the writer identification method proposed in the present invention. DETAILED DESCRIPTION

[0053] The invention is further described in detail with reference to the following specific examples and accompanying drawings. The processes, conditions, experimental methods, etc. for implementing the present invention, except for those specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0054] The present invention provides a writer identification method based on the fusion of time and space features. The following is a detailed introduction to the workflow of the method of the present invention, taking the real scene of calligraphy practice in daily life as the specific background; the part of the Chinese characters written in the specific implementation process of the present invention is Figure 1 (Only part of the content is shown).

[0055] Step 1. Dataset Construction: Use intelligent hardware such as an intelligent handwriting dot matrix pen to collect the temporal trajectory data of students practicing calligraphy on the dot matrix copybook at fixed intervals (determined by the characteristics of the hardware device itself. Different hardware devices have different sampling frequencies and sampling intervals, generally at the millimeter level) and upload them to the database to form the original dataset. The information contained in the original data structure is shown in Table 1, and the writing dynamic information contained in the sampling points is shown in Table 2.

[0056] Table 1 Original data structure information

[0057]

[0058] Table 2 Data information included in the sampling points

[0059]

[0060]

[0061] Step 2: Handwritten Chinese character image conversion: Convert the trajectory time series data in the original dataset into a two-dimensional handwritten Chinese character image. The specific steps are as follows:

[0062] Step i. Sampling point segmentation: All data points are segmented according to the pen tip motion information of each sampling data point. Pen tip motions include PEN_DOWN (pen tip touches the paper), PEN_MOVE (pen tip moves on the paper), and PEN_UP (pen tip leaves the paper). From PEN_DOWN to PEN_UP is a complete stroke;

[0063] Step ii. Information extraction: Extract the spatial position coordinates (ab_x and ab_y), pen tip force information (force), and timestamp information (timeLong) from the segmented sampling data points. Other useless information is discarded.

[0064] Step iii. Stroke connection: Connect the spatial coordinates of all adjacent sampling points within the same stroke. Determine the thickness of the connection line based on the average pen tip force between two sampling points. The greater the average pen tip force, the thicker the connection line. After connecting all the strokes, a complete Chinese character is formed.

[0065] Step iv. Saving and naming: Save the drawn handwritten Chinese character image, and the name includes the primary key of the image and the identifier of the writer. For example, '60a0cd975656e172c615b89b-6-状-65'. The first part represents the primary key of the copybook line, which serves as the unique identifier of the copybook line in the copybook. The second part represents the space serial number of the copybook line. The third part represents the Chinese character corresponding to the copybook line. The fourth part represents the identity label of the writer. The first three parts form the primary key of the handwritten Chinese character image.

[0066] The converted handwritten Chinese character image is as Figure 1 shown.

[0067] Image preprocessing: Preprocess the converted handwritten Chinese character image. The specific steps are as follows:

[0068] Step A. Image cleaning: Since the dataset has not been systematically processed, there are duplicate images and blurred images in the generated handwritten Chinese character images. Duplicate images and blurred images need to be removed, and those images that are clear but lack strokes should be retained because the lack of strokes does not affect the extraction of the writer's identity space.

[0069] Step B. Image scaling: Uniformly scale the pictures to a size of 128×128. The purpose of scaling is to adapt to subsequent image processing operations. First, read the height H and width W of the input image, and then calculate the corresponding scaling ratio:

[0070]

[0071] where scale_H and scale_W are the target image sizes, that is, the expected image sizes after scaling. Next, select the smaller value ratio of ratio_H and ratio_W as the appropriate image scaling ratio and perform the scaling:

[0072] new_H = int(ratio×H)

[0073] new_W = int(ratio×W)

[0074] Step C. Image filling: During the image scaling process, there may be an error between the target image size and the scaled image size. Calculate the vertical and horizontal offsets between the two:

[0075] dy = int(scale_H - new_H)

[0076] dx = int(scale_W - new_W)

[0077] Then generate random offset values in the vertical and horizontal directions:

[0078] dy=randint(0,dy)

[0079] dx=randint(0,dx)

[0080] Generate a blank image of size scale_H×scale×W, and fill the scaled image at the [dy:dy+new_H,dx:dx+new_W] position of the blank image.

[0081] Image dataset division: In a specific embodiment, the preprocessed image dataset is evenly divided into five sub-datasets to facilitate subsequent five-fold cross-validation, and the size of each sub-dataset is 5688.

[0082] Spatial representation extraction network construction: choose GR-RNN network [9] As a benchmark recognition network (after multiple comparative experiments, it has been proved that GR-RNN has the best spatial representation extraction ability among different convolutional neural networks), the model structure of this network is as follows Figure 2 As shown in Figure 1, this network combines the advantages of CNNs and RNNs. The CNN consists of four convolutional layers. An attention mechanism is added after the convolutional block of the fourth convolutional layer (experiments have shown that adding this attention mechanism after the deepest convolutional block in the recognition network has the best effect on recognition), forming a spatial representation extraction network. The GR-RNN contains four convolutional blocks, each consisting of a 3×3 convolutional layer, a BatchNorm layer, a ReLU layer, and a MaxPool layer. The input is a 128×128×1 black-and-white image. The output feature maps after the four convolutional blocks are f1: 128×128×64, f2: 64×64×128, f3: 32×32×256, and f4: 16×16×512. A global average pooling layer (GAP) is then used to convert f4 into a global context feature f4. This f4 is then split into multiple fragment sequences, which are then fed into a residual recurrent neural network along with the global context feature to integrate the global and partial representations.

[0083] like Figure 3 As shown, the workflow of the attention mechanism in the present invention is as follows:

[0084] Step a. Multi-granularity feature extraction: Three sparse convolution kernels of sizes 3×3, 5×5, and 7×7 are applied to the 16×16×512 output X of the convolutional layer of the fourth convolutional block of the recognition network. This yields three feature maps F1, F2, and F3 of the same size. These feature maps are spatial features of different granularities extracted from the handwritten Chinese character images.

[0085] Step b. Multi-granularity feature fusion: The feature maps of different granularities are concatenated according to the channel dimension to obtain a feature map of size 16×16×1536. The channel is then compressed into the original fused feature map F of size 16×16×512 using a 1×1 convolution kernel to achieve adaptive dynamic fusion of spatial features of different granularities.

[0086] Step c. Weight calculation: By performing average pooling on the spatial dimension, the average value of the fusion feature map F in the channel dimension is calculated to obtain a channel global feature representation vector S of size 1×1×C. The fully connected layer is used to compress the channel of S to obtain a vector Z of size 1×1×d. Then, for each feature map of different granularity, a separate fully connected layer is used to convert Z into corresponding weights. These weights represent the importance of spatial features of different granularity in the final fusion. Finally, the Softmax activation function is used to normalize all weights so that the weight of each feature map sums to 1 in the channel dimension.

[0087] Step d. Weighted summation: Perform weighted summation on each feature map and the corresponding weight to obtain the final fusion feature V.

[0088] The network structure of GR-RNN is as follows Figure 2 As shown, the specific attention mechanism structure is as follows Figure 3 As shown in the figure, this attention mechanism has two unique features. One is to extract multi-granularity spatial features by using sparse convolution kernels of different sizes, and the other is to adaptively fuse the extracted multi-granularity spatial features by using convolution kernels of size 1.

[0089] Writer spatial representation extraction: The handwritten Chinese character image is input into the GR-RNN network (the attention mechanism described above is introduced after the deepest convolution block of the GR-RNN network), and the final output result logits is used as the writer identity spatial representation.

[0090] Step 3: 2D time series feature map conversion: Assume that the information of a certain dimension (including ab_x, ab_y and force) in the original trajectory time series data is X=(x t ,t=1,2,…,N), the original trajectory data is converted into a two-dimensional time series feature map using the relative position matrix method. Assuming a time series X=(x t ,t=1,2,…,N), the specific steps are as follows:

[0091] Step I. Sequence normalization. For the original sequence, a standard normal distribution is obtained by z-score normalization method.

[0092] Where μ represents the mean of the time series X and σ represents the standard deviation of X.

[0093] Step II. Sequence dimensionality reduction. Using the segmented aggregation PAA method, set the scaling factor k to 2 (the scaling factor determines the degree of detail preservation. In order to retain more detail information in the writing trajectory, the scaling factor is given a smaller value). A new smoothed sequence is generated by the following formula Reduce the original dimension N to m:

[0094]

[0095] By calculating the average value of the piecewise constants to reduce the dimension, the approximate trend of the original time series can be maintained, and the resulting smooth time series The length is m.

[0096] Step III. Relative position calculation. Calculate the relative position between two timestamps and smooth the preprocessed time series. Convert to a two-dimensional matrix M:

[0097]

[0098] Step IV. Normalization. Use maximum and minimum normalization to convert M into a grayscale value matrix to obtain the final relative position matrix F:

[0099]

[0100] Step V. Concatenate the time series feature maps obtained by converting the ab_x, ab_y and force sequences by channel to obtain a three-channel time series feature and normalize it to 128×128. The converted three-channel time series feature map is as follows: Figure 4 shown.

[0101] Writer temporal representation extraction: The improved GR-RNN network is selected as the temporal representation extraction network (multiple comparative experiments have shown that among different convolutional neural networks, GR-RNN has the best temporal representation extraction capability, and the improvement method is the same as the spatial representation extraction network). The converted two-dimensional three-channel temporal feature map is input into the temporal representation extraction network, and the final output result logits is obtained as the temporal representation of the writer's identity.

[0102] Step 4: Fusion of temporal and spatial features: Match the spatial representation and temporal representation obtained by converting the same writing trajectory time series data, and perform post-fusion. The fusion method used here is to directly add the predicted probabilities of the spatial dimension and the temporal dimension, that is, the final output logits, as the final writer identity. The writer's identity is determined based on this representation. The complete recognition method process is as follows: Figure 5 shown.

[0103] Tables 3 and 4 show a comparison of the recognition accuracy of the method of the present invention and other recognition methods. The copied characters are created by the writer following the gray character template on the copybook. Due to the limited writing trajectory, the recognition accuracy on this data is lower. As shown in the table, the writer recognition method based on spatiotemporal feature fusion achieves higher Top 1 and Top 5 accuracy rates for all data types compared to traditional recognition methods.

[0104] Table 3. Comparison of overall recognition accuracy

[0105]

[0106] Table 4 Comparison of recognition accuracy between copied characters and ordinary characters

[0107]

[0108] In addition, the attention mechanism added to the spatial representation extraction network can better demonstrate its advantages compared to ordinary convolution kernels when the sparsity of handwritten pixels is greater. The experimental results are shown in Table 5, where 100% is the size of the original data set. It can be seen from the table that when the size of the data set is gradually reduced proportionally, the decrease in the spatial dimension Top 1 recognition accuracy of the sparse convolution kernel is less than the decrease in the spatial dimension Top 1 recognition accuracy of the ordinary convolution kernel, and the smaller the data size, the more obvious this trend is. The experimental results prove that the present invention not only has higher accuracy than previous recognition methods, but also has stronger robustness.

[0109] Table 5 Comparison of overall recognition accuracy on datasets of different sizes

[0110]

[0111] In the context of contract authenticity verification, the proposed writer identification method based on spatiotemporal feature fusion can be used to confirm the identity of handwritten signatures or signed content. Unlike traditional methods that rely solely on static signature glyph comparison, the proposed method collects the dynamic writing trajectory during contract signing (including stroke sequence, force changes, writing speed, etc.) to construct a comprehensive identity feature vector that combines spatial representation (glyph) and temporal representation (dynamic features). This vector is used to compare historical signature samples, effectively identify forged signatures or proxy signing, and enhance the authenticity verification capability of the contract signing process.

[0112] In exam scenarios, the method of the present invention can be embedded in electronic answering devices to collect and identify each candidate's handwriting in real time. The system can convert the candidate's handwriting trajectory during the answering process into an image and time-series feature graph, extract the candidate's personalized writing behavior characteristics, and compare them with handwriting features pre-collected during the exam registration process, for example, to identify whether the registered candidate is answering the questions and prevent cheating by proxy. Compared to traditional image comparison-based solutions, the method of the present invention incorporates dynamic features, making it more difficult to forge and more secure.

[0113] Identifying the attribution of large numbers of handwritten documents is a common challenge in large-scale archival management or the digitization of handwritten records. The proposed method can visualize and model the handwriting trajectory data of historical documents, extracting the spatial and temporal characteristics of the document's writer. This data is then compared with handwriting feature vectors of known writers in a database, achieving high-precision writer attribution classification.

[0114] References

[0115] [1]Purohit N,Panwar S.State-of-the-art:offline writer identificationmethodologies[C] / / 2021international conference on computer communication andinformatics(ICCCI).IEEE,2021:1-8.

[0116] [2]Chen Z, Yu HX, Wu A, et al. Level online writer identification[J]. International Journal of Computer Vision, 2021, 129(5): 1394-1409.

[0117] [3]Tu Y,Lin W,Mak M W.A survey on text-dependent and text-independentspeaker verification[J].IEEE Access,2022,10:99038-99049.

[0118] [4]Ramezani R.A language-independent authorship attribution approachfor author identification of text documents[J].Expert Systems withApplications,2021,180:115139.

[0119] [5]Morris R N.Forensic handwriting identification:fundamentalconcepts and principles[J].2020.

[0120] [6]Chen L,Chen P,Lin Z.Artificial intelligence in education:A review[J].Ieee Access,2020,8:75264-75278.

[0121] [7]Hameed M M,Ahmad R,Kiah M L M,et al.Machine learning-based offlinesignature verification systems:A systematic review[J].SignalProcessing:ImageCommunication,2021,93:116139.

[0122] [8]Sharifani K,Amini M.Machine learning and deep learning:A review ofmethods and applications[J].World Information Technology and EngineeringJournal,2023,10(07):3897-3904.

[0123] [9] He S, Schomaker L. GR-RNN: Global-context residual recurrent neural networks for writer identification[J]. Pattern Recognition, 2021,117:107975.

[0124] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal translation scripting language JavaScript, etc.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0128] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0129] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

[0130] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the appended claims.

Claims

1. A writer identification method based on spatiotemporal feature fusion, characterized in that: The method adaptively extracts spatial representation and temporal representation from handwriting trajectory data, and performs writer identification after fusing the representations. The method includes the following steps: Step 1: Collect the writer's real-time writing data and construct the original data set; Step 2: Convert the trajectory data in the original dataset into handwritten Chinese character images, and extract the writer's spatial representation from the handwritten Chinese character images; Step 3: Convert the trajectory data in the original dataset into a time series feature graph, and extract the writer's time representation from the time series feature graph; Step 4: Match and fuse the spatial representation and temporal representation obtained from the same writing trajectory data as the final writer identity representation, and determine the writer's identity based on the representation.

2. The identification method according to claim 1, wherein: In step one, the writing data is collected through an intelligent hardware device, and the writing data includes spatial feature data of Chinese characters and dynamic time series data during the writing process.

3. The identification method according to claim 1, wherein: In step 2, the conversion of trajectory data into handwritten Chinese character images includes the following steps: Segment all sampling data points according to strokes, extract valid information of sampling data points in each segment, connect the spatial coordinates of all adjacent sampling data points in the same stroke with line segments, determine the thickness of the connecting line segments, and complete the handwritten Chinese character image drawing; and / or, Save the drawn handwritten Chinese character image.

4. The identification method according to claim 3, wherein: The valid information of the sampling data point includes: spatial position coordinates, pen tip force information, and timestamp information; and / or, The pen tip strength of two adjacent sampling points determines the thickness of the connecting line segment; and / or, The handwritten Chinese character image is saved and named; the name includes an image primary key and a writer identifier; the image primary key includes a copybook row primary key, a copybook row space sequence number, and the Chinese character corresponding to the copybook row.

5. The identification method according to claim 1, wherein: In step 2, the handwritten Chinese character image is preprocessed including image cleaning, image scaling, and image filling before extracting spatial features; The image cleaning includes removing duplicate and blurred images from the handwritten Chinese character image; the image scaling includes scaling different handwritten Chinese character images to the same size; the image filling includes filling the scaled image in a blank image of target size; and / or, Extract spatial representations from Chinese character images using convolutional neural networks; and / or, The convolutional neural network is a GR-RNN network; and / or, An attention mechanism is added after the convolutional block of the fourth convolutional layer in the GR-RNN network to form a spatial representation extraction network to extract the spatial representation of the writer corresponding to the handwritten Chinese character image; and / or, The spatial representation extraction network extracts features through multi-granularity sparse convolution, fuses feature maps of different granularities, calculates weights and performs weighted summation to obtain the writer's spatial representation.

6. The identification method according to claim 5, wherein: The preprocessed image data is divided into five sub-datasets and five-fold cross validation is performed.

7. The identification method according to claim 1, wherein: In step 3, the original trajectory data is converted into a two-dimensional time series feature map using the relative position matrix method: The original trajectory data in a time series is subjected to sequence standardization, sequence dimension reduction, relative position calculation and normalization operations to generate three time series feature maps of different dimensions. After splicing and conversion according to the channel dimension, a three-channel two-dimensional time series feature map is obtained; and / or, The GR-RNN temporal representation extraction network outputs the writer's temporal representation.

8. The identification method according to claim 1, wherein: In step 4, the writer's identity is comprehensively judged by combining the spatial representation obtained from the static writing results and the temporal representation obtained from the dynamic writing process through feature fusion operations.

9. An identification system for implementing the identification method according to any one of claims 1 to 8, characterized in that: The recognition system includes: a data acquisition module, a data conversion module, a feature extraction module, a feature fusion module, and a recognition module; The data acquisition module is used to collect the real-time writing data of the writer; The data conversion module is used to convert the collected trajectory data into handwritten Chinese character images and time series feature maps; The feature extraction module is used to extract spatial features and temporal features from the image and the time series feature map respectively; The feature fusion module is used to fuse spatial features and temporal features to generate a writer's identity representation; The recognition module is used to determine the identity of the writer based on the fused representation.

10. Application of the recognition method according to any one of claims 1 to 8, or the recognition system according to claim 9, in fields including contract authenticity verification, exam cheating prevention, and handwritten document classification.

Images