Calligraphy image recognition method and system based on computer vision algorithm

Abstract

The invention discloses a calligraphy image recognition method and system based on a computer vision algorithm, and relates to the technical field of calligraphy image recognition, and the method comprises the steps: collecting dynamic trajectory data through a high-speed camera device and a pressure-sensitive pen, and generating a dynamic trajectory matrix; a multispectral imager is used for obtaining a material characteristic pattern of paper fiber deformation and ink mark penetration, and fusion data is generated through affine transformation alignment; inputting the fusion data into an LSTM-CNN network to extract spatio-temporal features, outputting a dynamic stroke segmentation map, and separating material codes, technique features and style latent variables from the fusion data through a three-channel decoupling network; carrying out preliminary classification, carrying out secondary classification on characters with similar shapes in combination with an attention mechanism, outputting a fine-grained classification result, carrying out weighted fusion on the classification result, carrying out style verification and track logicality, and outputting identification tags and style attributes; and misjudgment samples trigger incremental learning of the laminated classification model, and the weight of a secondary classifier is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of calligraphy image recognition, and in particular to a calligraphy image recognition method and system based on computer vision algorithms. Background Art

[0002] In recent years, with the rapid development of computer vision and deep learning technologies, the field of calligraphy image recognition has gradually shifted from traditional static image analysis to multi-modal perception and semantic understanding of the dynamic writing process. For example, in scenarios that require high-precision analysis of writing styles, when calligraphy learners are copying, they need to know whether the pen movement force and speed conform to the target style; for example, in the scenario of personalized font generation, designers write basic glyphs, and the system extracts style latent variables (such as connecting stroke habits), and combines style interpolation to generate a mixed font library, saving manual design time. Early calligraphy recognition systems were mainly based on optical character recognition (OCR) technology, which realized character classification by extracting stroke morphology and structural features. However, its reliance on the characteristics of single RGB images results in insufficient ability to capture writing dynamic information (such as pen tip trajectory, pen movement pressure) and material characteristics (such as paper fiber deformation, ink penetration depth). With the progress of sensor technology, the introduction of pressure-sensitive pens, high-speed cameras, and multi-spectral imagers has made it possible to synchronously collect dynamic trajectory data (pen tip coordinates, acceleration, inclination) and multi-spectral material characteristics (infrared reflectivity, polarization scattering characteristics). On this basis, researchers have tried to combine temporal models (such as LSTM) with spatial feature extraction networks (such as ResNet) to build a spatio-temporal fusion framework to improve recognition accuracy. However, existing methods still have significant bottlenecks in cross-modal data alignment, feature decoupling optimization, and incremental learning adaptability.

[0003] The main deficiencies of the existing technology are reflected in that the heterogeneity of dynamic trajectories and multi-spectral data leads to insufficient spatio-temporal alignment accuracy, and traditional affine transformation and Kalman filtering methods are difficult to eliminate clock drift between devices and non-linear deviations of coordinate systems. For example, the cumulative error between the local coordinates of the pressure-sensitive pen and the global mapping will destroy the trajectory continuity. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a calligraphy image recognition method based on computer vision algorithms to solve the problem that the heterogeneity of dynamic trajectories and multi-spectral data leads to insufficient spatio-temporal alignment accuracy, and traditional affine transformation and Kalman filtering methods are difficult to eliminate clock drift between devices and non-linear deviations of coordinate systems.

[0006] To solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a calligraphy image recognition method based on computer vision algorithms, which includes:

[0008] Collect dynamic trajectory data through a high-speed camera device and a pressure-sensitive pen to generate a dynamic trajectory matrix, obtain a material feature map of paper fiber deformation and ink penetration using a multispectral imager, and generate fused data through affine transformation alignment;

[0009] Input the fused data into an LSTM-CNN network to extract spatio-temporal features, output a dynamic stroke segmentation map, and separate material encoding, technique features, and style latent variables from the fused data through a three-channel decoupling network;

[0010] Use ResNet-50 for preliminary classification, use DenseNet-121 combined with an attention mechanism for secondary classification of similar-shaped characters, analyze the starting angle and stroke topology differences, output a fine-grained classification result, generate a virtual style image based on the material encoding and style latent variables, verify the consistency with the original Figure 1 Consistency, weighted fusion of the classification result, style verification, and trajectory logic, and output an identification label and style attribute;

[0011] Misjudged samples trigger incremental learning of the stacked classification model, generate mixed training data through style interpolation, and optimize the weights of the secondary classifier.

[0012] As a preferred solution of the calligraphy image recognition method based on computer vision algorithms according to the present invention, wherein: the specific steps for generating the fused data are as follows:

[0013] Collect the pen tip coordinates, pressure, and speed through the pressure-sensitive pen to construct a five-dimensional dynamic trajectory matrix;

[0014] Use a multispectral imager to perform normal incidence diffuse reflection, transmission, and polarization scanning to output a multispectral image, and extract material characteristics from the multispectral image through an image processing algorithm;

[0015] Optimize the coordinate mapping parameters based on the reference point coordinates through the recursive least squares method, and compensate for the spatio-temporal deviation of the device in combination with an adaptive Kalman filter to generate spatio-temporally synchronized HDF5 fused data.

[0016] As a preferred solution of the calligraphy image recognition method based on computer vision algorithms according to the present invention, wherein: separating the material encoding, technique features, and style latent variables from the fused data specifically includes:

[0017] Input the fused data into an LSTM-CNN network, extract the trajectory time series features through bidirectional LSTM, extract the material space features using the improved ResNet-34, and splice the trajectory time series features and the material space features;

[0018] After splicing spatio-temporal features, dilated spatial pyramid pooling is used to generate a dynamic stroke segmentation map;

[0019] The material encoding, technique features and style latent variables are separated from the dynamic stroke segmentation map through a three-channel decoupling network.

[0020] As a preferred scheme of the calligraphy image recognition method based on computer vision algorithm described in the present invention, wherein: the specific steps of outputting the recognition label and style attribute are as follows:

[0021] ResNet-50 is used for coarse classification, and DenseNet-121 combined with a spatial pyramid attention module is used for fine classification of similar Chinese characters;

[0022] Subcategories are divided based on the calculation of the starting pen angle difference from the trajectory starting point coordinates;

[0023] Virtual style images are generated through a variational autoencoder, and the multi-spectral channel SSIM and KL divergence are calculated to verify the consistency with the original, Figure 1 and the final label is output by weighted fusion of the classification confidence, style consistency and trajectory logic weights.

[0024] As a preferred scheme of the calligraphy image recognition method based on computer vision algorithm described in the present invention, wherein: the misjudged samples trigger incremental learning of the stacked classification model, specifically:

[0025] When misjudged samples trigger incremental learning, deformable residual incremental blocks are stacked on the top of DenseNet-121;

[0026] Mixed training data is generated through spherical interpolation of style latent variables, and abnormal samples are filtered by combining a stroke order validator.

[0027] As a preferred scheme of the calligraphy image recognition method based on computer vision algorithm described in the present invention, wherein: the incremental learning refers to:

[0028] Incremental learning adopts two-stage optimization. In the first stage, the parameters of the basic classifier are frozen and the Wasserstein distance is minimized to align the feature distributions;

[0029] In the second stage, when jointly fine-tuning, a trajectory-style consistency loss term is added, and the overfitting parameters are dynamically rolled back through an online verification loop.

[0030] As a preferred scheme of the calligraphy image recognition method based on computer vision algorithm described in the present invention, wherein: the specific steps of generating mixed training data through style interpolation and optimizing the weights of the secondary classifier are as follows:

[0031] The feature extraction weights are dynamically adjusted through deformable residual incremental blocks, and the deformable residual incremental blocks are stacked on the top of the DenseNet-121 secondary classifier;

[0032] Calculate the importance matrix of the base classifier parameters using the elastic weight consolidation algorithm, and impose L2 regularization constraints on the key weights to limit the gradient update amplitude;

[0033] Generate mixed training data based on spherical interpolation of style latent variables, and filter out abnormal samples that violate the stroke logic through a stroke order validator;

[0034] In the two-stage optimization, the base classifier parameters are frozen in the first stage and the Wasserstein distance is minimized to align the feature distributions. In the second stage, a trajectory-style consistency loss term is added during joint fine-tuning to force the model to maintain the relevance between the stroke timing logic and the style features;

[0035] Dynamically detect the overfitting tendency through an online verification loop and roll back the parameters to improve the model's generalization ability for misjudged samples.

[0036] In a second aspect, the present invention provides a calligraphy image recognition system based on a computer vision algorithm, including: an acquisition module, an extraction module, a verification module, an optimization module, and a fusion module:

[0037] The acquisition module is responsible for the efficient acquisition and spatio-temporal alignment of high-precision dynamic trajectory data and multi-spectral material features;

[0038] The extraction module is responsible for fusing dynamic trajectories and material features to extract deep spatio-temporal correlated representations;

[0039] The verification module realizes fine-grained classification of calligraphy images and improves the accuracy through multi-modal consistency verification;

[0040] The optimization module is used to perform online model optimization for misjudged samples to improve the system robustness;

[0041] The fusion module is used to integrate multi-modal data (such as text, images), generate unified representations using complementary and redundant information, and a logical verification module verifies the consistency and detects anomalies.

[0042] In a third aspect, the present invention provides a computer device, including a memory and a processor, where: the memory stores a computer program, and: when the computer program is executed by the processor, any step of the calligraphy image recognition method based on the computer vision algorithm as described in the first aspect of the present invention is implemented.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and: when the computer program is executed by the processor, any step of the calligraphy image recognition method based on the computer vision algorithm as described in the first aspect of the present invention is implemented.

[0044] The beneficial effects of the present invention are as follows: Through atomic clock synchronization and adaptive Kalman filter compensation, the present invention eliminates clock drift and coordinate mapping errors between devices, enabling microsecond-level synchronous alignment between the dynamic trajectory and the paper fiber characteristics, improving data acquisition accuracy. Material-technique-style decoupled modeling: Using a three-channel decoupling network to separate material encoding, technique features, and style latent variables, and combining with the multi-spectral Retinex algorithm to extract paper fiber deformation characteristics, breaking through the limitations of traditional single-feature extraction and enhancing the fine-grainedness of stroke segmentation. Dynamic threshold classification of similar-shaped characters: Extracting spatio-temporal features through an LSTM-CNN network, switching the sampling frequency in combination with the acceleration threshold, and constructing a secondary classification tree based on the starting pen angle and stroke topology differences, improving the recognition accuracy of similar-shaped characters. Misjudgment sample incremental optimization mechanism: Adopting a deformable residual incremental block and style interpolation generation technology, and combining with the elastic weight consolidation algorithm to protect core parameters, enabling the model to dynamically learn new samples after misjudgment without destroying the original knowledge system and solving the problem of catastrophic forgetting. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0046] Figure 1 It is the architecture diagram of multi-modal data acquisition and spatio-temporal synchronization in Embodiment 1.

[0047] Figure 2 It is the schematic diagram of cross-modal stitching in Embodiment 1.

[0048] Figure 3 It is the flow chart of three-channel feature decoupling and classification decision-making in Embodiment 1.

[0049] Figure 4 It is the schematic diagram of style interpolation in Embodiment 1. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the following will make a detailed description of the specific embodiments of the present invention with reference to the drawings in the specification.

[0051] Many specific details are set forth in the following description in order to provide a thorough understanding of the present invention. However, the present invention may be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0052] Second, the "one embodiment" or "embodiment" referred to herein means a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The appearances of "in one embodiment" in different places in this specification do not all refer to the same embodiment, nor are they separate or alternative embodiments that are mutually exclusive of other embodiments.

[0053] Embodiment 1, referring to Figures 1 to 4 , is the first embodiment of the present invention. This embodiment provides a calligraphy image recognition method based on computer vision algorithms, including the following steps:

[0054] S1. Collect dynamic trajectory data through a high-speed camera device and a pressure-sensitive pen to generate a dynamic trajectory matrix; use a multispectral imager to obtain a material feature map of paper fiber deformation and ink penetration, and generate fused data through affine transformation alignment;

[0055] Make a reference point array using infrared reflective materials at the four corners and the center of the writing plane. Each reference point is embedded with a unique ID code. The high-speed camera is equipped with an infrared filter. Before writing starts, modulated infrared light is emitted to activate the reference points, and the three-dimensional coordinates of each reference point are automatically identified through phase detection. When the pressure-sensitive pen first touches the paper surface, an infrared synchronization signal generator is triggered to lock the clock sources of the camera and the pressure-sensitive pen to the same atomic clock, ensuring that the timestamp synchronization error is less than the microsecond level. The first contact here marks the starting point of the trajectory, and the coordinates of the trajectory starting point are determined by the initial contact position of the pressure-sensitive pen;

[0056] Use a biaxial laser rangefinder to project a cross-shaped positioning laser on the writing plane, and construct a plane equation based on the reference point coordinates:

[0057] Z = ax + by + d;

[0058] Where, Z represents the dependent variable, representing the depth of the writing plane in three-dimensional space, which is the result calculated by the linear combination of x and y. a and b represent the weights of x and y on Z. x and y represent the abscissa and ordinate in the coordinate system, and d is a constant term (intercept), representing the value of Z when x = 0 and y0;

[0059] The pressure-sensitive pen is built-in with a gyroscope to output inclination data in real time. The nib coordinates (x ′ , y ′ ) are mapped to the global coordinate system through the coordinate transformation formula. The global coordinate system is a unified spatial reference system calibrated by the reference point coordinates. The local coordinate system of the pressure-sensitive pen is mapped to the global coordinate system through an inclination-dependent affine transformation. The expression is:

[0060] x = x'cosθ + y'sinθ + Δx;

[0061] y = -x'sinθ + y'cosθ + Δy;

[0062] Among them, x and y represent the abscissa and ordinate in the local coordinate system of the pressure-sensitive pen, x' and y' represent the abscissa and ordinate in the global coordinate system, θ represents the pen tip inclination angle (unit: degree), Δx and Δy are the reference point correction offsets, cos and sin represent the cosine function and sine function, and the recursive least squares method is used to update the transformation parameters online. Parameter optimization is performed once every time a reference point traversal is completed;

[0063] A three-dimensional coordinate system is established using a biaxial laser rangefinder, and the local coordinate system of the pressure-sensitive pen is mapped to the global coordinate system of the camera. An adaptive Kalman filter is used to compensate for the spatio-temporal deviation between the two devices. The dynamic adjustment formula for the filtering parameters is:

[0064]

[0065] Among them, § represents the calibration coefficient, ΔD represents the fluctuation coefficient of the pressure-sensitive pen coordinate measurement value (unit: mm), which is obtained by calculating the position standard deviation of 100 consecutive sampling points and reflects the accuracy error of the device itself. ν represents the instantaneous movement speed of the pen tip (unit: mm / s), which is calculated by dividing the coordinate difference between two adjacent frames by the time interval, and Δ v represents the standard deviation of the camera speed calculation (unit: mm / s), represents the instantaneous pressure value of the pressure-sensitive pen, ε represents the anti-zero correction amount, and the value is one percent of the order of magnitude of Δ v to ensure that the denominator is not zero;

[0066] The original data of the pressure-sensitive pen is processed by a sliding window. The acceleration data of the pressure-sensitive pen during regular writing (such as uniform pen movement) and rapid movements (such as pressing, lifting, turning) is collected, and the standard deviation is calculated as the reference. Based on the acceleration mean within the sliding window ± 3 times the standard deviation (3σ principle), the switching threshold is set. The default setting in the regular mode is set to 100Hz (10ms / frame), which is applicable to the stable writing state; in the event-driven mode, it is shortened to 200Hz (5ms / frame) to capture rapid pen tip changes (such as pen stops, flying white);

[0067] When it is detected that the acceleration exceeds the switching threshold, it switches to the event-driven mode, shortens the sampling interval to half of the regular mode, collects the acceleration through the sensor, and constructs a five-dimensional matrix based on the pen tip inclination angle, movement coordinates, instantaneous movement speed, and acceleration to obtain the dynamic trajectory matrix:

[0068]

[0069] Among them, represents the acceleration, and M t represents the dynamic trajectory matrix M at time t;

[0070] Perform three - level scanning on paper fiber deformation and ink penetration using a multispectral imager. The three - level scanning refers to the synchronous scanning in the spatial dimension and the spectral dimension (wavelength), including normal - incidence diffuse - reflection scanning, transmission scanning, and polarization scanning;

[0071] It should be noted that for normal - incidence diffuse - reflection scanning, an annular LED array is used to irradiate at different wavelengths to collect the reflection spectral characteristics of paper fibers. For transmission scanning, a uniform backlight is set on the back of the paper to quantify the ink penetration depth distribution. For polarization scanning, linearly polarized light is incident at a 45° angle, and the microscopic morphology of the ink surface is obtained by rotating the analyzer;

[0072] Use optical elements such as filters, gratings, or prisms to separate the mixed light into discrete narrow - band wavelengths according to the wavelength. For example, a filter array (such as a linear - gradient filter) or grating spectroscopy technology can project light of different wavelengths onto different regions of the detector;

[0073] After spectral separation, the energy distribution of each band is converted into an electrical signal by a detector (such as a CCD or CMOS) to form the original image of each band;

[0074] Adopt an improved Retinex algorithm to perform Gaussian convolution on the original images of each band to obtain the low - frequency illumination component. According to the original images of each band and the low - frequency illumination component, the reflection component is obtained by separating in the logarithmic domain, and the reflection component characterizes the inherent properties of the material (such as chemical composition, surface texture), that is, the material feature map;

[0075] Use a biaxial laser rangefinder to project a cross - positioning laser, construct a plane equation based on the reference - point coordinates, map the local coordinate system of the pressure - sensitive pen to the global coordinate system through rotation (based on the tip inclination angle θ) and translation amounts (Δx, Δy), and use the recursive least - squares method to update the transformation parameters online. Perform parameter optimization every time a reference - point traversal is completed to ensure the real - time accuracy of coordinate mapping;

[0076] Compensate for the spatio - temporal deviation between the high - speed camera and the pressure - sensitive pen through an adaptive Kalman filter, and dynamically adjust the filtering parameters (such as the fluctuation coefficient Δ C 、instantaneous motion rate ν, instantaneous pressure value ) through the dynamic adjustment formula of the filtering parameters to improve the robustness of the trajectory data;

[0077] The spatio - temporal deviation refers to the inconsistency between the high - speed camera and the pressure - sensitive pen in time synchronization and spatial - coordinate mapping:

[0078] Time deviation: The misalignment of the acquisition timestamps caused by clock drift between devices (which has been resolved to the microsecond level through atomic - clock synchronization).

[0079] Spatial deviation: Coordinate offsets caused by coordinate - system conversion errors (such as affine - transformation residuals) and sensor noise. For example:

[0080] The rotation and translation errors between the local coordinate system and the global coordinate system of the pressure-sensitive pen, and the registration error between the cross-positioning light spot projected by the laser rangefinder and the reference point;

[0081] The multi-spectral image is registered to the trajectory spatio-temporal coordinate system through the thin plate spline interpolation model TPS, and the preconditioned conjugate gradient method is used for iterative solution, and finally stored as fusion data in HDF5 format containing time synchronization indexes.

[0082] S2. Input the fusion data into the LSTM-CNN network to extract spatio-temporal features, output the dynamic stroke segmentation map, and separate the material encoding, technique features, and style latent variables from the fusion data through a three-channel decoupling network;

[0083] The fusion data in HDF5 format is split into a dynamic trajectory matrix (time series) and a material feature map (spatial image), and normalized respectively;

[0084] The dynamic trajectory matrix is divided into sliding windows according to the time step (100Hz / 200Hz), and each window contains 50 consecutive sampling points, forming a (number of samples × 50 × 5) tensor (5 dimensions: x, y, θ, ν, ); The material feature map is uniformly scaled to 256 × 256 pixels through bilinear interpolation, and the multi-spectral channel information (such as RGB + infrared + polarization) is retained to form a (number of samples × 256 × 256 × 6) tensor;

[0085] Based on the time synchronization index, the dynamic trajectory window is aligned with the material feature map frame at the corresponding moment to construct a spatio-temporal fusion tensor (number of samples × 50 × 256 × 256 × 11), where the 11 channels contain 5-dimensional trajectory data and 6-dimensional material data. The sliding stitching strategy is adopted to retain the trajectory continuity in the time dimension and keep the material texture intact in the spatial dimension;

[0086] Input the five-dimensional dynamic trajectory matrix into a bidirectional LSTM network, set the number of hidden units to 128, output the feature vector (dimension 128) at each time step, and weight and fuse the features of key time points (such as pen-down and turning points) through the Time Attention Gate to generate a trajectory encoding vector (dimension 256);

[0087] Input the material feature map into an improved ResNet-34 network (removing the fully connected layer), introduce a Channel Attention Block after the third residual block, calculate the weight coefficients of each spectral channel, enhance the response in the fiber deformation and ink penetration areas, and output a spatial feature map (size 32 × 32 × 512);

[0088] The trajectory encoding vector (256 - dimensional) is copied and expanded into a 32×32×256 spatial grid, which is concatenated with the material - space feature map (512 - dimensional) along the channel dimension to form a 768 - dimensional fused feature map. Cross - modal feature interaction is performed through a 3×3 convolutional kernel, and a 768 - dimensional fused tensor is output;

[0089] The fused feature map output by LSTM - CNN is input into an improved DeepLabv3+ decoder. Atrous Spatial Pyramid Pooling (ASPP) is used to capture stroke features at different scales. The atrous rates are set to 6, 12, and 18 respectively, corresponding to the feature extraction of the pen - tip starting point (small - range details), connected - stroke path (medium - range correlations), and overall stroke shape (large - range semantics);

[0090] The low - level detail features (such as paper fiber edges and microscopic ink deformations) output by the second residual block of ResNet - 34 are concatenated with the high - level semantic features (stroke topological structures), and a preliminary segmentation heat map is generated through 3×3 convolution;

[0091] The trajectory encoding vector (256 - dimensional) output by LSTM is spatio - temporally aligned with the preliminary segmentation heat map to form a spatio - temporal constraint tensor. The trajectory encoding vector contains the temporal information of the pen - tip coordinates, speed, and acceleration, such as the change in the movement direction of the pen - tip within 5 consecutive frames;

[0092] Based on the probability values of the preliminary segmentation heat map (the probability that each pixel belongs to a stroke or the background), which reflects the single - frame segmentation confidence, combined with the motion parameters in the trajectory encoding vector (such as the acceleration change rate), the label consistency weight of the same spatial position between adjacent frames is defined. For example, if the pen - tip speed suddenly drops at time t (a pen - pressing action), the labels at the same position in frames t and t + 1 should be forced to be consistent to avoid misjudging as a broken stroke;

[0093] The smoothing intensity of the Conditional Random Field (CRF) is dynamically adjusted through the instantaneous pressure value in the trajectory encoding vector. For example, when it is detected that increases (heavy pen - pressing), the constraint weight of label transfer between adjacent frames is increased to suppress over - segmentation in the ink - penetration area;

[0094] The mean - field approximation algorithm is used for 10 - iteration inference. Each iteration performs message passing, compatibility transformation, and output mapping. Specifically:

[0095] Based on the timestamp index in the trajectory encoding vector, the segmentation results of adjacent frames are propagated along the time axis (for example, the end of the stroke in frame t-1 constrains the starting position of frame t). The transition matrix learned through LSTM trajectory encoding is used to correct logical errors in label jumps between frames (for example, prohibiting the "pen down" label from appearing immediately after the "pen lift" action). The optimized CRF probability distribution is back-projected into the spatial dimension to generate a dynamic stroke segmentation map. Each pixel in the dynamic stroke segmentation map contains continuous labels along the time axis (such as the stroke start, stroke movement, turning point, and stroke end).

[0096] Material-related features (latter 384 dimensions) are separated from the fused feature map by compressing it into a 256-dimensional vector through global average pooling. This is then fed into a fully connected layer to generate a material encoding. A contrastive learning strategy is then employed, using the NT-Xent loss function to bring the encoding distances of similar paper samples closer and those of different paper samples further apart.

[0097] Extract technique-related features from the first 384-dimensional features. Specifically, focus on the brush tip motion area through the spatial attention mechanism to generate a 256-dimensional technique vector. Set a dynamic threshold control mechanism to activate the classification head of different technique modes (such as center / side) according to the instantaneous pressure value. Specifically, activate different technique classification branches dynamically according to the pressure value, for example:

[0098] When the instantaneous pressure value exceeds the dynamic technique switching threshold, the feature extraction branch of the "side edge" mode is activated;

[0099] When the instantaneous pressure value is less than the dynamic technique switching threshold, the feature extraction of the "center forward" mode is enabled;

[0100] Based on the calibration data of the pressure-sensitive pen (e.g., 1000 writing samples), the pressure distribution range corresponding to different technique modes (center tip / side tip) is calculated. For example, the average pressure of the center tip is 0.3N (standard deviation σ = 0.05) and the average pressure of the side tip is 0.6N (standard deviation σ = 0.1). The dynamic technique switching threshold can be set to the intersection of the two distributions (e.g., 0.45N);

[0101] The material code and technique features are concatenated and input into a variational autoencoder. The encoder outputs the mean and variance of the style latent variable (each dimension is 32). The style latent variable (dimension is 64) is sampled. The style latent variable refers to the low-dimensional vector representing the writing style encoded by the variational autoencoder. The decoder reconstructs a virtual style image through a deconvolutional network and verifies pixel-level consistency with the original calligraphy image collected by the multispectral imager. The KL divergence and reconstruction loss are optimized.

[0102] Construct three discriminator networks (material discriminator / technique discriminator / style discriminator), which respectively receive the concatenated input of the corresponding encoded vector and the encoded information of other channels. Through adversarial training, each channel is forced to discard redundant information. For example, the material discriminator receives the material encoding and technique features, outputs the correlation score between the two, and minimizes the score to decouple the material and the technique;

[0103] Adaptive adjustment of the adversarial loss weight according to the training stage. For example: in the initial stage, focus on feature reconstruction (weight ratio 4:1), in the middle stage, enhance the decoupling constraint (weight ratio 1:1), and in the later stage, fine-tune the balance (weight ratio 2:1) to avoid excessive suppression of a certain channel.

[0104] S3. Use ResNet-50 for preliminary classification, and use DenseNet-121 combined with the attention mechanism to perform secondary classification on similar-shaped characters, analyze the starting pen angle and stroke topology differences, output the fine-grained classification results, generate virtual style images based on the material encoding and style latent variables, and verify the consistency with the original Figure 1 Consistency, weighted fusion of classification results, style verification, and trajectory logic, and output the recognition label and style attributes;

[0105] Use the pre-trained ResNet-50 as the backbone network, remove the original fully connected layer, and replace it with two layers of adaptive fully connected layers. The first layer compresses the output dimension to 1 / 4 of the original feature space, and the second layer maps to the coarse-grained classification categories (such as the major categories of Chinese character structures: left-right structure and up-down structure);

[0106] Insert a channel attention mechanism after the third residual block of ResNet-50, assign higher weights to the feature channels in the stroke density area, and suppress background interference;

[0107] Use the Sharpness-Aware Minimization (SAM) optimization algorithm to replace the traditional Stochastic Gradient Descent (SGD). By simultaneously minimizing the loss value and the sharpness of the loss surface, improve the generalization ability of the ResNet-50 network model for similar-shaped characters. Freeze the parameters except the fully connected layer in the initial stage of training, and unfreeze all layers for end-to-end fine-tuning in the later stage;

[0108] Perform cross-modal concatenation on the coarse classification feature map output by ResNet-50 and the dynamic trajectory matrix (including trajectory coordinates, instantaneous movement speed, and instantaneous pressure value) to form spatio-temporal fusion features;

[0109] Introduce a spatial pyramid attention module after the transition layer of DenseNet-121 to capture the subtle differences at the stroke intersections through multi-scale pooling (1x1, 3x3, 5x5);

[0110] Use the Gated Recurrent Unit (GRU) to extract the temporal dependencies from the dynamic trajectory data and output the stroke coherence weights;

[0111] The outputs of the two branches are weighted and fused before being fed into the classification head. A fine-grained classification tree is constructed for similar characters (e.g., “未” and “末”), and subcategories are divided based on the difference in starting angles (calculated using the coordinates of the starting point of the trajectory).

[0112] The material code (characterizing the distribution of paper fibers) and technical features (such as the distribution of edge pressure) are input into the variational autoencoder (VAE). The encoder outputs latent variables that follow a Gaussian distribution, and the decoder generates a virtual style image through deconvolution.

[0113] Calculate the structural similarity index (SSIM) of a multispectral image in multiple spectral channels (RGB + infrared). The multispectral image includes a virtual style image and the original image, and identifies the difference in ink penetration depth. The expression is:

[0114]

[0115] SSIM 单通道 =l·c·s;

[0116] Among them, I g and I o Represent the virtual style image and the original image, μ g and μ o Represents the pixel mean of the virtual style image and the original image, σ g and σ o They represent the standard deviations of the virtual style image and the original image, C1, C2, and C3 represent constants, which are obtained by multiplying the corresponding weight coefficients according to the pixel value range. The corresponding weight coefficients can be set according to the requirements. l represents the brightness term, which measures the average brightness difference between the two images. c represents the contrast term. s represents the structure term. l(I g ,I o ) represents the brightness item of the virtual style image and the original image, c(I g ,I o ) represents the comparison term between the virtual style image and the original image, s(I g ,I o ) represents the structural term of the virtual style image and the original image, SSIM 单通道 Represents the structural similarity index of a single channel;

[0117] SSIM is calculated for each channel (R, G, B, infrared) and weighted averaged according to channel importance. For example, the infrared channel weight is 0.4 and the RGB channel weight is 0.2. The expression is:

[0118] SSIM 综合 =0.2(SSIM R +SSIM G +SSIM B )+0.4·SSIM 红外 ;

[0119] Among them, SSIM 综合 represents the structural similarity index SSIM that synthesizes the R, G, and B channels;

[0120] Input the virtual style image into the LSTM network to extract the time series features of stroke drawing (such as the order of pen tip rising and falling, and the change of pen movement direction), and output it as a time series feature vector. Compare the KL divergence between the time series feature vector and the acceleration distribution of the original trajectory matrix. If the comprehensive score of SSIM and KL divergence is lower than the dynamic threshold (automatically adjusted according to the statistical value of the training set), trigger the manual review process;

[0121] Set the classification confidence weight, style consistency weight, and trajectory logic weight;

[0122] The classification confidence weight is based on the Softmax output entropy values of ResNet-50 and DenseNet-121. The lower the entropy value (the higher the confidence), the higher the weight of the style consistency weight;

[0123] The style consistency weight is determined by the linear combination score of SSIM and KL divergence. The higher the score (the better the style consistency), the higher the weight;

[0124] The trajectory logic weight is the probability score calculated by the CRF model for the stroke time series coherence. The higher the score (the more reasonable the logic), the higher the weight;

[0125] Adopt a weighted voting mechanism. If the voting results of the three conflict, preferentially adopt the category with the highest trajectory logic weight. Input the style latent variable into the pre-trained StyleGAN latent space to retrieve the closest historical calligrapher style label (such as "Yan style", "Liu style"), output the style similarity percentage, and obtain the style label and style attributes.

[0126] S4. The misjudged samples trigger the incremental learning of the stacked classification model, generate mixed training data through style interpolation, and optimize the weights of the secondary classifier.

[0127] Deploy a confidence-logic joint discriminator at the classification result output end to monitor the dispersion degree of the three weights (classification confidence, style consistency, and trajectory logic) in real time;

[0128] When the difference between any two of the three weights exceeds the dynamic balance interval (for example, the difference between the confidence weight and the trajectory logic weight exceeds the balance threshold);

[0129] When the deviation of the style similarity percentage from the historical statistical mean of the classification label exceeds the standard deviation range;

[0130] When the difference in the ink penetration depth distribution between the virtual style image and the original image breaks through the material feature tolerance threshold;

[0131] Adopt a sliding window statistical mode. If there are more than three consecutive logical conflicts within the window, the misjudgment process is activated, and the spatio-temporal context data at the time of triggering is recorded (including the acceleration mutation points of the dynamic trajectory matrix and the coordinates of the polarization reflection abnormal areas in the multi-spectral image);

[0132] Stack a deformable residual increment block on top of the original DenseNet-121 secondary classifier. The deformable residual increment block consists of three sub-structures, including an elastic memory unit, a dynamic feature distillation layer, and an adversarial perturbation suppressor. Specifically:

[0133] The elastic memory unit adopts a gated convolution structure to retain the memory strength of the learned features in the historical classifier, and controls the fusion ratio of new and old knowledge through a trainable gating coefficient;

[0134] The dynamic feature distillation layer performs channel-level knowledge distillation on the deep feature map of the original classifier and the new features extracted by the increment block, and uses cosine similarity to constrain the consistency of the feature space distribution;

[0135] The adversarial perturbation suppressor injects adversarial noise samples before feature fusion to enhance the robustness of the increment module to abnormal features in misjudged samples;

[0136] A two-way gradient path is established between the deformable residual increment block and the base classifier. During forward propagation, new features are preferentially extracted through the increment block, and the sampling position of the convolution kernel is changed by learning the offset, enabling the network to adaptively adjust the actual receptive field of each neuron according to the input material characteristics. The receptive field refers to the area range on the input image that a certain neuron on the feature map of a certain layer in the convolutional neural network can "see". It determines the range of input spatial information that the neuron can capture, enhancing the adaptability to misjudged samples. The offset learning mechanism of DCN allows the feature sampling points to change with the input, improving the flexibility of feature extraction. During backward propagation, through the gradient masking mechanism, according to the parameter importance matrix (calculated by the elastic weight consolidation algorithm), a mask is applied to the core weights of the base classifier to limit the update amplitude of the core weight gradient of the base classifier. Through the skip connection of ResNet, a reverse path is established between the increment block and the base classifier, allowing the gradient to be backpropagated from the increment block to the base layer, but the backpropagation intensity is controlled by the gating coefficient (such as a weight of 0.5);

[0137] Use a variational autoencoder to extract style latent variables and material encodings from misjudged samples, and construct a mixed data generation pool according to the style latent variables and material encodings. Specifically:

[0138] Use a pre-trained variational autoencoder (VAE) to extract style latent variables and material encodings from misjudged samples. The style latent variables are mapped to the latent space through the encoder network to represent the low-dimensional distribution of writing styles (such as "Yan style" or "Liu style"), and the material encodings represent the deformation of paper fibers and the ink penetration characteristics;

[0139] In the latent space of the variational autoencoder, concatenate the style latent variables and the material encodings into joint latent variables, and reduce the dimension to three-dimensional space through principal component analysis to construct a material-style hyperplane to constrain the semantic continuity of the interpolation path;

[0140] Select the latent variable endpoints of two different style labels (such as "Yan style" and "Liu style") and perform spherical linear interpolation on the hyperplane;

[0141] According to the feature dispersion of the misjudged samples, calculate through the covariance matrix of the latent variable distribution, and add isotropic Gaussian noise to the interpolation points to generate enhanced latent variables;

[0142] Input the enhanced latent variables into the decoder network of the variational autoencoder to reconstruct multi-spectral virtual images and corresponding dynamic trajectory matrices to form a mixed data generation pool;

[0143] Set a two-stage optimizer. Specifically, in the first stage, freeze the parameters of the base classifier and only train the incremental module: use the elastic weight consolidation algorithm to calculate the importance matrix of the base classifier parameters, impose quadratic constraints on the key weights, introduce the feature distribution alignment term between the style interpolation samples and the original samples in the loss function, and minimize the Wasserstein distance W between them in the high-dimensional space;

[0144] In the second stage, start joint fine-tuning and adopt an adaptive learning rate allocation mechanism. Specifically: dynamically allocate the learning rate according to the gradient activity of the parameters in the historical training, apply momentum acceleration to the low-frequency updated parameters, and add a trajectory-style consistency loss term to the optimization objective to force the classifier to maintain the relevance between the stroke timing logic and the style features during the update process;

[0145] Embed an online verification loop during the optimization process. After each batch of parameter updates, use the validation set to detect the model offset. When the overfitting tendency is detected, automatically roll back to the parameter snapshot of the nearest stable state.

[0146] This embodiment also provides a calligraphy image recognition system based on computer vision algorithms, including: a collection module, an extraction module, a verification module, an optimization module, and a fusion module:

[0147] The collection module is responsible for the efficient collection and spatio-temporal alignment of high-precision dynamic trajectory data and multi-spectral material features;

[0148] The extraction module is responsible for fusing dynamic trajectories and material features to extract deep spatio-temporal correlated representations;

[0149] The verification module realizes fine-grained classification of calligraphy images and improves accuracy through multi-modal consistency verification;

[0150] The optimization module is used to perform online model optimization for misjudged samples to improve the robustness of the system;

[0151] The fusion module is used to integrate multi-modal data (such as text, images), generate unified representations using complementary and redundant information, and the logic verification module verifies the consistency and detects anomalies.

[0152] This embodiment also provides a computer device applicable to the case of a calligraphy image recognition method based on computer vision algorithms, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the calligraphy image recognition method based on computer vision algorithms as proposed in the above embodiment.

[0153] This computer device can be a terminal. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a carrier network, NFC (Near Field Communication), or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, a touchpad, or a mouse, etc.

[0154] This embodiment also provides a storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the calligraphy image recognition method based on computer vision algorithms as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, abbreviated as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, abbreviated as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, abbreviated as EPROM), programmable read-only memory (Programmable Red-Only Memory, abbreviated as PROM), read-only memory (Read-Only Memory, abbreviated as ROM), magnetic memory, flash memory, magnetic disk or optical disc.

[0155] In summary, the present invention achieves the following: atomic clock synchronization and adaptive Kalman filter compensation to eliminate clock drift and coordinate mapping errors between devices, enabling microsecond-level synchronous alignment of dynamic trajectories and paper fiber features, improving data acquisition accuracy; decoupled modeling of material-technique-style: using a three-channel decoupling network to separate material encoding, technique features, and style latent variables, and combining the multi-spectral Retinex algorithm to extract paper fiber deformation features, breaking through the limitations of traditional single-feature extraction and enhancing the fine-grainedness of stroke segmentation; dynamic threshold classification of similar-shaped characters: extracting spatio-temporal features through an LSTM-CNN network, switching the sampling frequency in combination with an acceleration threshold, and constructing a secondary classification tree based on the starting pen angle and stroke topology differences to improve the recognition accuracy of similar-shaped characters; incremental optimization mechanism for misjudged samples: adopting a deformable residual incremental block and style interpolation generation technology, and combining the elastic weight consolidation algorithm to protect core parameters, enabling the model to dynamically learn new samples after misjudgment without destroying the original knowledge system and solving the problem of catastrophic forgetting.

[0156] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A calligraphy image recognition method based on computer vision algorithms, characterized in that: Including: Collect dynamic trajectory data through a high-speed camera device and a pressure-sensitive pen to generate a dynamic trajectory matrix. Use a multispectral imager to obtain a material feature map of paper fiber deformation and ink penetration, and generate fused data through affine transformation alignment; Input the fused data into an LSTM-CNN network to extract spatio-temporal features, output a dynamic stroke segmentation map, and separate the material encoding, technique features, and style latent variables from the fused data through a three-channel decoupling network; Use ResNet-50 for preliminary classification, and use DenseNet-121 combined with an attention mechanism to perform secondary classification on similar-shaped characters. Analyze the starting pen angle and stroke topology differences, output a fine-grained classification result, generate a virtual style image based on the material encoding and style latent variables, verify the consistency with the original image, and weighted-fuse the classification result, style verification, and trajectory logic to output the recognition label and style attributes; Misjudged samples trigger incremental learning of the stacked classification model, generate mixed training data through style interpolation, and optimize the weights of the secondary classifier.

2. The calligraphy image recognition method based on computer vision algorithm according to claim 1, characterized in that: The specific steps for generating the fused data are as follows: Collect the pen tip coordinates, pressure, and speed through a pressure-sensitive pen to construct a five-dimensional dynamic trajectory matrix; Use a multispectral imager to perform normal incidence diffuse reflection, transmission, and polarization scans to output a multispectral image, and extract material characteristics from the multispectral image through an image processing algorithm; Optimize the coordinate mapping parameters based on the reference point coordinates through the recursive least squares method, and compensate for the spatio-temporal deviation of the device by combining an adaptive Kalman filter to generate spatio-temporally synchronized HDF5 fused data.

3. The calligraphy image recognition method based on computer vision algorithm according to claim 2, wherein: The specific method for separating the material encoding, technique features, and style latent variables from the fused data is as follows: Input the fused data into an LSTM-CNN network, extract the trajectory temporal features through a bidirectional LSTM, and use an improved ResNet-34 to extract the material spatial features, and splice the trajectory temporal features and the material spatial features; After splicing the spatio-temporal features, use atrous spatial pyramid pooling to generate a dynamic stroke segmentation map; Separate the material encoding, technique features, and style latent variables from the dynamic stroke segmentation map through a three-channel decoupling network.

4. The method for calligraphy image recognition based on computer vision algorithm according to claim 3, characterized in that: The specific steps for outputting the recognition label and style attributes are as follows: Use ResNet-50 for coarse classification, and use DenseNet-121 combined with a spatial pyramid attention module for fine classification of similar-shaped characters; Calculate the difference in the starting pen angle based on the coordinates of the starting point of the trajectory to divide subcategories; Generate a virtual style image through a variational autoencoder, calculate the SSIM and KL divergence of the multispectral channels to verify the consistency with the original image, and weighted-fuse the classification confidence, style consistency, and trajectory logic weights to output the final label.

5. The calligraphy image recognition method based on computer vision algorithm according to claim 4, characterized in that: The misjudged samples triggering incremental learning of the stacked classification model are specifically as follows: When misjudged samples trigger incremental learning, deformable residual incremental blocks are stacked on top of DenseNet-121; Generate mixed training data through spherical interpolation of style latent variables, and filter out abnormal samples in combination with a stroke sequence validator.

6. The calligraphy image recognition method based on computer vision algorithm according to claim 5, wherein: The incremental learning refers to: Incremental learning adopts two-stage optimization. In the first stage, the parameters of the basic classifier are frozen and the Wasserstein distance is minimized to align the feature distributions; During the second-stage joint fine-tuning, a trajectory-style consistency loss term is added, and the overfitting parameters are dynamically rolled back through an online verification loop.

7. The method for calligraphy image recognition based on computer vision algorithm according to claim 6, characterized in that: The hybrid training data is generated by style interpolation to optimize the weights of the quadratic classifier. The specific steps are as follows: The feature extraction weights are dynamically adjusted through deformable residual increment blocks, and the deformable residual increment blocks are stacked on top of the DenseNet-121 quadratic classifier; The elastic weight consolidation algorithm is used to calculate the importance matrix of the base classifier parameters, and L2 regularization constraints are imposed on the key weights to limit the gradient update amplitude; Hybrid training data is generated based on spherical interpolation of style latent variables, and abnormal samples that violate the stroke logic are filtered through a stroke order validator; In the two-stage optimization, the parameters of the base classifier are frozen in the first stage, and the Wasserstein distance is minimized to align the feature distributions. During the second-stage joint fine-tuning, a trajectory-style consistency loss term is added to force the model to maintain the correlation between the stroke timing logic and the style features; The overfitting tendency is dynamically detected through an online verification loop, and the parameters are rolled back to improve the generalization ability of the model for misjudged samples.

8. A calligraphy image recognition system based on computer vision algorithms, based on the calligraphy image recognition method based on computer vision algorithms according to any one of claims 1 to 7, characterized in that: Including: An acquisition module, an extraction module, a verification module, an optimization module, and a fusion module: The acquisition module is responsible for the efficient acquisition and spatio-temporal alignment of high-precision dynamic trajectory data and multi-spectral material features; The extraction module is responsible for fusing the dynamic trajectory and material features and extracting deep spatio-temporal correlated representations; The verification module realizes the fine-grained classification of calligraphy images and improves the accuracy through multi-modal consistency verification; The optimization module is used to perform online model optimization for misjudged samples to improve the robustness of the system; The fusion module is used to integrate multi-modal data (such as text, images), generate a unified representation using complementary and redundant information, and the logical verification module verifies the consistency and detects anomalies.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that: When the processor executes the computer program, the steps of the calligraphy image recognition method based on the computer vision algorithm according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the calligraphy image recognition method based on the computer vision algorithm according to any one of claims 1 to 7 are implemented.

Images