A Chinese character recognition method and system based on multi-modal feature fusion

Abstract

The application relates to the technical field of Chinese character recognition, and discloses a Chinese character recognition method and system based on multi-modal feature fusion, which comprises the following steps: firstly, a gray image of a Chinese character to be recognized is acquired, skeleton and distance features are extracted, and a center line is fitted; secondly, edge asymmetry, gray centroid normal offset and half-width dissipation features are calculated based on gray, gradient and distance features respectively, and are fused into local non-symmetrical dissipation energy flow features, a weighted integration is carried out to obtain a string non-symmetrical dissipation energy flow moment to lock a normal direction; then, an anisotropy measurement is constructed according to the edge asymmetry feature, and the image is resampled in a string coordinate system; then, the string core interval is determined by combining the energy flow moment and a monotonic change criterion; subsequently, a string mask is generated according to a bandwidth ratio, energy compensation is carried out on the resampled image to obtain a standard character form; finally, the standard character form is matched with a standard character form library to complete recognition, so that high-precision and robust Chinese character recognition is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Chinese character recognition, more particularly, it relates to a Chinese character recognition method and system based on multi-modal feature fusion. BACKGROUND

[0002] In the field of recognition of Chinese characters in running and cursive scripts, the complexity of character images far exceeds that of printed or regular scripts. Running and cursive scripts are written at a fast speed, with frequent pen stroke connection and large ink color variation, and often have visual structures such as silk connection, white space, heavy ink, and partial stroke. Such handwriting is not independent and closed within a single character, but presents a silk-like connection extending outward from the main stroke. Due to differences in writing pressure, stroke direction, and paper ink absorption performance, the gray scale distribution and edge sharpness of these connection lines are significantly non-uniform. Especially under unstable scanning or shooting conditions, there are a large number of fine line structures in running and cursive images between real handwriting and pseudo-connection, which brings great difficulty to the stable division of character boundaries.

[0003] Traditional running and cursive recognition methods generally rely on stroke segmentation, contour extraction, or axis tracking to describe the handwriting form. These methods often assume that the character contour is symmetrical, continuous, and has a smooth thickness variation, which is suitable for printed or standard writing. However, in running and cursive images, the stroke thickness varies dramatically, and a stroke is often accompanied by a speed change resulting in a sharp outer edge and a blurred inner edge. The axis extraction algorithm is prone to deviation in such areas, and the contour normal direction frequently flips with the sudden changes in thickness, making the edge, gray scale, and geometric parameters show contradictory information on different image channels. For example, the end position of the same stroke appears as a break point on the skeleton graph, but still shows continuity on the gray scale graph; conversely, the false connection caused by ink penetration is mistaken for real handwriting. This phenomenon makes it impossible for traditional methods to form a stable geometric coordinate system in the key stroke transition area, thereby weakening the accuracy and robustness of subsequent recognition.

[0004] Between the last stroke and the adjacent stroke of running and cursive writing, there is often a fine silk structure formed by the combined action of speed increase, pressure reduction, and paper capillary diffusion. This structure appears as a thin band with gradually decaying gray scale and thickness on the image, and its outer boundary is usually sharper than the inner boundary, resulting in a significant asymmetry in the image gradient and brightness distribution on the left and right sides. Due to the different physical formation mechanisms of the region, the stroke width, gray scale, and edge sharpness do not change synchronously in different spatial directions, and the normal section established by conventional algorithms cannot maintain uniqueness. As a result, at the transition between the end of the stroke and the silk, the image's outside and inside definitions will flip with minor changes in noise or resolution, causing the boundary line, skeleton line, and width curve to be misaligned with each other, and the recognition system to produce systematic deviations at this point. SUMMARY

[0005] The application provides a Chinese character recognition method and system based on multi-modal feature fusion, and solves the technical problem that the normal section of the stroke transition area lacks uniqueness in a cursive Chinese character image, causing the labels on the outside and the inside to drift in space, and thus the recognition calculation based on the stroke form lacks a stable geometric reference system.

[0006] In a first aspect, a Chinese character recognition method based on multi-modal feature fusion comprises:

[0007] S1, a gray image of a Chinese character to be recognized is acquired, a skeleton of each pixel width in the gray image is extracted, and a distance feature of each pixel to the nearest boundary is extracted, candidate branches of a pulling silk are screened according to the skeleton, and a center line is fitted for the candidate branches of the pulling silk;

[0008] S2, edge asymmetry features on the center line are calculated based on gradient features of the gray image, gray centroid normal offset features on the center line are calculated based on gray features of the gray image, and half-width dissipation features on the center line are calculated based on the distance features;

[0009] The edge asymmetry features, the gray centroid normal offset features and the half-width dissipation features are fused to obtain local non-symmetry dissipation energy flow features on the center line, the local non-symmetry dissipation energy flow features are weighted and integrated along the position of the center line to obtain a pulling silk non-symmetry dissipation energy flow moment, and the normal direction is locked according to the local non-symmetry dissipation energy flow features in the initial segment of the center line;

[0010] S3, an anisotropy metric is constructed based on the edge asymmetry features and a preset amplification coefficient, and the gray image is resampled in a pulling silk coordinate system defined by the anisotropy metric;

[0011] S4, the local non-symmetry dissipation energy flow features are smoothed, and a dissipation energy flow threshold determined based on the pulling silk non-symmetry dissipation energy flow moment and a monotonic variation criterion of the local non-symmetry dissipation energy flow features are combined to determine a pulling silk core interval;

[0012] S5, a pulling silk mask is generated in the pulling silk coordinate system according to a preset bandwidth ratio, and energy compensation along the tangential direction is performed on the resampled gray image through the pulling silk mask to obtain a standard character form;

[0013] S6, an identification result is obtained based on the standard character form and a standard character form library.

[0014] In a second aspect, a Chinese character recognition system based on multi-modal feature fusion is applied to any one of the Chinese character recognition methods based on multi-modal feature fusion, and comprises:

[0015] A data acquisition module acquires a gray image of a Chinese character to be recognized, extracts a skeleton of each pixel width in the gray image and a distance feature of each pixel to the nearest boundary, screens candidate branches of a pulling silk according to the skeleton, and fits a center line for the candidate branches of the pulling silk.

[0016] a feature extraction module, which calculates an edge asymmetry feature on the center line based on a gradient feature of a grayscale image, calculates a grayscale centroid normal offset feature on the center line based on a grayscale feature of the grayscale image, and calculates a half-width dissipation feature on the center line based on a distance feature;

[0017] a feature fusion calculation module, which fuses the edge asymmetry feature, the grayscale centroid normal offset feature, and the half-width dissipation feature to obtain a local non-symmetry dissipation energy flow feature on the center line, weights and integrates the local non-symmetry dissipation energy flow feature along the position of the center line to obtain a string non-symmetry dissipation energy flow moment, and locks a normal direction according to the local non-symmetry dissipation energy flow feature in a starting segment of the center line;

[0018] a resampling module, which constructs an anisotropy metric based on the edge asymmetry feature and a preset magnification coefficient, and resamples the grayscale image in a string coordinate system defined by the anisotropy metric;

[0019] an interval calculation module, which smoothes the local non-symmetry dissipation energy flow feature, and determines a string core interval in combination with an energy flow threshold value determined based on the string non-symmetry dissipation energy flow moment and a monotonic variation criterion of the local non-symmetry dissipation energy flow feature;

[0020] a standard character shape generation module, which generates a string mask in the string coordinate system according to a preset bandwidth ratio, and performs tangential energy compensation on the resampled grayscale image through the string mask to obtain a standard character shape;

[0021] a character shape recognition module, which obtains a recognition result based on the standard character shape and a standard character shape library.

[0022] The present application has the beneficial effect that by fusing multi-modal features such as grayscale, gradient, and distance, a stable geometric reference system is constructed, effectively solving the problems of boundary drift and unstable recognition caused by complex structures such as stroke connection, white space, and false connection in running and cursive script Chinese characters. By introducing the local non-symmetry dissipation energy flow feature and the string coordinate system, the energy distribution and direction change of the stroke ending and connected string transition zone can be accurately described. Through the feature fusion and resampling mechanism, the system can automatically restore the true geometric shape of the handwriting without relying on manual segmentation, significantly improving the standardization accuracy and recognition robustness of the character shape. Compared with traditional recognition methods that only rely on a single image channel or symmetric assumptions, the present application has stronger noise resistance and adaptability when processing running and cursive script and irregular handwriting, thereby greatly improving the accuracy and stability of overall Chinese character recognition. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a flowchart of a Chinese character recognition method based on multi-modal feature fusion of the present application;

[0024] Figure 2 is a framework diagram of a Chinese character recognition system based on multi-modal feature fusion according to the present application. DETAILED DESCRIPTION

[0025] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is merely meant to provide a better understanding of the subject matter described herein and can include changes, modifications, additions or omissions of the functions and arrangements of the elements discussed without departing from the scope of the present specification. Various examples can omit, substitute, or add various processes or components in addition to those described or in lieu thereof. Also, features described in relation to some examples can also be combined in other examples.

[0026] Embodiment One

[0027] As shown in Figure 1 , a Chinese character recognition method based on multi-modal feature fusion includes:

[0028] S1, obtaining a gray image of a Chinese character to be recognized, extracting a skeleton of each pixel width in the gray image and a distance feature of each pixel to the nearest boundary, screening a candidate branch of a pulling silk according to the skeleton, and fitting a center line of the candidate branch of the pulling silk;

[0029] S2, calculating an edge asymmetry feature on the center line based on a gradient feature of the gray image; calculating a gray centroid normal offset feature on the center line based on a gray feature of the gray image; and calculating a half-width dissipation feature on the center line based on the distance feature;

[0030] Fusing the edge asymmetry feature, the gray centroid normal offset feature, and the half-width dissipation feature to obtain a local non-symmetry dissipation energy flow feature on the center line; weighting and integrating based on the position of the local non-symmetry dissipation energy flow feature along the center line to obtain a pulling silk non-symmetry dissipation energy flow moment; and locking a normal direction according to the local non-symmetry dissipation energy flow feature in a starting segment of the center line;

[0031] S3, constructing an anisotropy metric based on the edge asymmetry feature and a preset amplification coefficient, and resampling the gray image in a pulling silk coordinate system defined by the anisotropy metric;

[0032] S4, performing smoothing processing on the local non-symmetry dissipation energy flow feature, and determining a pulling silk core interval in combination with an energy flow threshold value determined based on the pulling silk non-symmetry dissipation energy flow moment and a monotonic variation criterion of the local non-symmetry dissipation energy flow feature;

[0033] S5, generating a pulling silk mask in the pulling silk coordinate system according to a preset bandwidth ratio, and performing tangential energy compensation on the resampled gray image through the pulling silk mask to obtain a standard character form;

[0034] S6 obtains the recognition results based on queries in the standard character set and the standard character set library.

[0035] In one embodiment of the present invention, a grayscale image of the Chinese character to be recognized is obtained; the skeleton of each pixel width in the grayscale image and the distance features from each pixel to the nearest boundary are extracted; candidate branches for connecting lines are selected based on the skeleton; and a center line is fitted to the candidate branches for connecting lines.

[0036] Obtain the grayscale image of the Chinese character to be recognized. ;

[0037] If the grayscale image has the th If the grayscale value of the nth pixel is less than or equal to the grayscale threshold, then the nth pixel will be... Each pixel is marked as 1, otherwise it is marked as 0, to obtain a binary image. ;

[0038] Determine the binary image Foreground boundary set ;in, Furthermore, there are pixels within the 8-neighborhood of each foreground boundary pixel. make ;

[0039] Get the The Euclidean distance between the coordinates of the nth pixel and the coordinates of each foreground boundary pixel is calculated, and the minimum Euclidean distance is selected as the nth pixel. Distance features per pixel ;

[0040] Define skeleton set Among them, the skeleton set There are two skeleton pixels in it. and satisfy: , so that:

[0041] ,in, Represents the first element in the skeleton set. One skeleton pixel, Indicates the first Distance features of each skeleton pixel, It is the Euclidean norm;

[0042] The first skeleton set The pixel width of each skeleton pixel is ;

[0043] skeleton set The maximum pixel width of all skeleton pixels in the array is denoted as . ,Will As the threshold for the width of the main stroke And define the main pen connected region. ;

[0044] In skeleton collection In the connected graph, the starting point is selected from the connected regions of the main stroke. path As a candidate branch for silk-drawing; among them. The arc length parameter of the path. Let the total arc length of the path be a parameter that satisfies:

[0045] Path length ;in, For path The first derivative, The minimum length threshold is preset.

[0046] Maximum pixel width of all pixels along the path except the starting point ;in, Maximum width limit;

[0047] The derivative of path width with arc length It holds true in more than 70% of the total path length; among which, The derivative is calculated using the three-point difference method to determine the monotonically varying thinning rate threshold. , The sampling step size is the arc length.

[0048] Collect candidate branches of the silk thread Fitted pixel set ;in, To fit the number of skeleton pixels in the pixel set, To fit the set of pixels, the first The pixel coordinates of each skeleton pixel;

[0049] Perform cubic spline fitting on the skeleton pixel set to minimize the energy function. Obtain the center line Wherein, the energy function is:

[0050] ;

[0051] in, To fit the skeleton pixels in the pixel set The corresponding arc length parameter, Centerline The second derivative, For smoothing weights.

[0052] A gray scale threshold is set. If the gray scale value of the i-th pixel is less than or equal to the threshold (indicating that the pixel belongs to the stroke), it is marked as 1; if the gray scale value is greater than the threshold (indicating that the pixel belongs to the background), it is marked as 0. Finally, a binary image containing only 0 (background) and 1 (foreground) is obtained.

[0053] The foreground boundary is a quantitative representation of the stroke edge, defined as: a pixel marked as 1 (foreground) in the binary image, and at least one of its 8 adjacent pixels (up, down, left, right, and four diagonal directions) is marked as 0 (background). All pixels that satisfy this condition form the foreground boundary set.

[0054] The distance feature is used to describe the distance of each pixel to the stroke edge: for the i-th pixel, calculate the straight-line distance (Euclidean distance) to all pixels in the foreground boundary set, and select the smallest distance value as the distance feature of the pixel. This feature can reflect the position of the pixel within the stroke, i.e., the larger the distance feature, the closer the pixel is to the center of the stroke.

[0055] The skeleton is the central axis of the stroke, which is a geometric reference for analyzing the morphology of the filament. The definition rule is: each pixel in the skeleton set (the p-th skeleton pixel) needs to find two different pixels in the foreground boundary set, such that the distance from the skeleton pixel to these two boundary pixels is exactly equal, and this distance value is exactly equal to the distance feature of the skeleton pixel. Thus, it is ensured that the skeleton pixel is always located at the geometric center of the stroke (with equal distance to the edges on both sides).

[0056] The stroke width is an indicator to distinguish the main pen from the filament: since the distance feature of the skeleton pixel is the distance to a single edge, the stroke width at the position of the skeleton pixel is equal to twice the distance feature (the sum of the distances to the edges on both sides), i.e., the pixel width of the p-th skeleton pixel = 2 x the distance feature of the pixel.

[0057] The main pen is thicker than the filament in handwriting, so: first find the maximum value of the pixel width of all skeleton pixels, and take this maximum value as the main pen width threshold (the minimum width standard of the main pen); then, divide all skeleton pixels in the skeleton set whose pixel width reaches the main pen width threshold according to connectivity (continuous regions formed by adjacent pixels), and the connected regions obtained are the main pen connected regions.

[0058] The filament is a thin connection between the main pens, which needs to be filtered from the connected graph of the skeleton set (continuous path network formed by skeleton pixels). The filtering rule needs to satisfy 4 conditions simultaneously:

[0059] Starting point condition: the starting end of the path must be located within the main pen connected region (ensuring that it is a branch extending from the main pen);

[0060] Length condition: total length of the path Need to be greater than or equal to the preset minimum length threshold (Exclude short noise twigs);

[0061] Width condition: the maximum pixel width of all skeleton pixels on the path except the starting point needs to be less than or equal to the maximum width upper limit ; wherein, (The width upper limit is 0.5 times the width of the main pen, because the silk is thinner than the main pen);

[0062] Tapering trend condition: the rate of change (derivative) of the path width along the arc length needs to be less than or equal to the negative monotonic tapering rate threshold , and this condition needs to be established in more than 70% of the total length of the path (the physical characteristics of the silk are that it gradually tapers outward from the main pen, so the width decreases with the increase of the length, the derivative is negative, and most of the length needs to meet this trend).

[0063] Wherein, the derivative of the path width along the arc length is calculated by three-point difference method: taking the width values of three consecutive points on the path, using the width difference between the latter point and the former point, divided by 2 times the arc length sampling step (the arc length distance between adjacent two points), to get the width change rate of the middle point.

[0064] The selected silk candidate branch is a discrete skeleton pixel, which needs to be fitted into a smooth continuous curve (center line) for feature calculation:

[0065] First step: select N skeleton pixels from the silk candidate branch to form a fitting pixel set, each pixel has a clear coordinate;

[0066] Second step: use cubic spline fitting method (an interpolation method that can ensure smooth curve), to get the center line by minimizing the energy function. The energy function includes two parts:

[0067] The sum of the squares of the distances from all fitting pixels to the corresponding positions on the center line (to ensure that the fitted curve is close to the original skeleton pixels, to ensure accuracy);

[0068] The integral of the second derivative of the center line multiplied by the smoothing weight (the second derivative reflects the bending degree of the curve, the integral term can control the smoothness of the center line, the larger the smoothing weight, the smoother the curve, to avoid distortion of the center line caused by discrete pixels). The final center line is the geometric carrier for silk shape analysis.

[0069] In one embodiment of the present application, the gradient feature based on the gray image is used to calculate the edge asymmetry feature on the center line; the gray feature based on the gray image is used to calculate the gray centroid normal offset feature on the center line; the distance feature is used to calculate the half-width dissipation feature on the center line, including:

[0070] On the center line Compute tangent vector at ;

[0071] where, is the arc length derivative of the centerline; is the centerline at arc length parameter ;

[0072] Define normal vector :

[0073] where, and are the lateral and longitudinal components of ;

[0074] Define half-width function based on distance feature , the pixel position of the normal profile arbitrary point is ; where, is the normal coordinate;

[0075] For gray image , the horizontal gradient and the vertical gradient are calculated by Sobel operator to obtain the spatial gradient vector ;

[0076] Compute the normal gradient derivative:

[0077] ;

[0078] Compute the inside maximum gradient absolute value:

[0079] ;

[0080] Compute the outside maximum gradient absolute value:

[0081] ;

[0082] The edge asymmetry feature is ;

[0083] Define the normal gray distribution ;

[0084] The gray centroid normal offset feature is calculated by trapezoidal numerical integration with a sampling step of 0.5 pixels ;

[0085] The half-width dissipation feature is calculated by three-point difference method with an arc length step of 1 pixel ; where, Arc length difference step for derivative calculation.

[0086] Tangent vector is used to describe the direction of the centerline at each position, and is the basis for establishing the local coordinate system (tangent-normal): for each position of the centerline, calculate its rate of change along the arc length direction (arc length derivative, reflecting the change of direction with length), and then through the Euclidean norm (normalization processing, ensuring that the vector length is 1, only keeping the direction information), get the tangent vector of this position. The vector can clearly extend the direction of the centerline.

[0087] Normal vector is perpendicular to the tangent vector, and is used to analyze the characteristics along the width direction of the centerline (perpendicular to the extension direction) (such as edge, gray distribution), and the definition rule is: if the transverse component of the tangent vector is tx and the longitudinal component is ty, then the transverse component of the normal vector is -ty and the longitudinal component is tx. Thus, the normal vector is perpendicular to the tangent vector, and the direction is unified.

[0088] Half-width function and normal profile are the key to limit the analysis range in the width direction:

[0089] Half-width function: based on distance feature definition, the half-width value of each position of the centerline is equal to the distance feature of that position, and the distance feature is the minimum distance from the pixel to the single-sided edge, and the distance feature at the centerline is the width to the single-sided edge;

[0090] Normal profile position: along the normal vector direction, taking the centerline position as the origin, the range of normal coordinate r is [-w(s), +w(s)] (covering the complete width from the single-sided edge to the other side edge), and the pixel position of any point in this range is equal to the coordinate of the centerline at that position plus the normal coordinate r multiplied by the normal vector, which clearly defines the pixel range to be analyzed in the width direction.

[0091] Gradient vector is used to describe the intensity and direction of gray scale change, and is the basis for extracting the sharpness of the edge: for a gray scale image, the horizontal gradient (gray scale change rate along the x-axis direction) and the vertical gradient (gray scale change rate along the y-axis direction) of each pixel are calculated using the Sobel operator (a commonly used edge detection operator that can effectively capture gray scale mutations), and the two gradient values are combined to obtain the spatial gradient vector of the pixel. The greater the magnitude of the vector, the more intense the gray scale change at the pixel (the sharper the edge).

[0092] Normal gradient derivative is the gray scale change rate along the width direction (normal direction), which is used to focus on the gray scale mutation at the edge: the spatial gradient vector is projected onto the normal vector direction to obtain the normal gradient derivative. The derivative reflects the gray scale change intensity along the width direction (from one side edge to the other side edge), that is, the absolute value of the normal gradient derivative will significantly increase at the edge position (gray scale mutation).

[0093] Inner and outer maximum gradient absolute value, used to quantify the sharpness difference between the two edges in the width direction:

[0094] Inner: the region with normal coordinate r < 0 (the side close to the main pen), in which the maximum absolute value of the normal gradient derivative is taken, denoted as inner maximum gradient absolute value;

[0095] Outer: the region with normal coordinate r > 0 (the side far from the main pen), in which the maximum absolute value of the normal gradient derivative is taken, denoted as outer maximum gradient absolute value. These two values represent the sharpness of the inner and outer edges, respectively, i.e., the larger the value, the sharper the edge.

[0096] Edge asymmetry feature is an index to quantify the sharpness difference between the inner and outer edges: the difference between the outer maximum gradient absolute value and the inner maximum gradient absolute value is divided by the sum of the outer maximum gradient absolute value and the inner maximum gradient absolute value to obtain the edge asymmetry feature. The value of this feature ranges from -1 to 1: a positive value indicates that the outer edge is sharper than the inner edge, and a negative value indicates that the inner edge is sharper than the outer edge, matching the property of the outer side of the calligraphy silk being sharp and the inner side being blunt.

[0097] Gray centroid normal offset feature is used to quantify the asymmetry of the gray distribution in the width direction (the gray distribution will be biased to the inner side due to the rapid attenuation of the gray scale on the outer side of the silk):

[0098] First step: define the normal gray distribution, i.e., the gray value at each position (corresponding to different r) on the normal profile;

[0099] Second step: use trapezoidal numerical integration (an integral approximation method suitable for discrete pixels, which calculates the area by regarding the gray values of adjacent pixels as the upper and lower bases of a trapezoid) to calculate two integrals:

[0100] The integral of the normal coordinate r multiplied by the corresponding gray value (reflecting the position sum weighted by gray);

[0101] The integral of the gray value (reflecting the total gray sum);

[0102] Third step: divide the first integral by the second integral to obtain the normal coordinate of the gray centroid, i.e., the difference between this coordinate and the center line (r = 0), which is the gray centroid normal offset feature. This feature reflects the offset trend of the gray in the width direction.

[0103] Half-width dissipation feature is used to quantify the thinning trend of the silk along the extension direction (arc length) (the core dissipation characteristic of the silk is to gradually thin out from the main pen):

[0104] First step: determine the arc length sampling step (fixed at 1 pixel to ensure uniform calculation accuracy);

[0105] Second step: calculate the derivative of the half-width function along the arc length direction by three-point difference method (calculate the change rate of the middle position by the half-width values of the continuous three arc length positions). The derivative reflects the change trend of the half-width along the arc length: a negative value indicates that the half-width decreases with the increase of the arc length (the filament becomes thinner), and a positive value indicates that the half-width increases (not in line with the characteristics of the filament);

[0106] Third step: the derivative is the half-width dissipation feature, which quantifies the dissipation degree of the filament thinning.

[0107] In an embodiment of the application, the local asymmetric dissipation energy flow feature on the center line is obtained by fusing the edge asymmetry feature, the gray centroid normal offset feature and the half-width dissipation feature, including:

[0108] The gray centroid normal offset feature is processed by non-dimensionalization to obtain the normalized offset feature ; wherein is a very small positive number;

[0109] The half-width dissipation feature is processed by one-way constraint and non-dimensionalization, including:

[0110] The positive part of the half-width dissipation is retained ;

[0111] The normalized positive dissipation feature is calculated;

[0112] The edge asymmetry feature, the normalized offset feature and the normalized positive dissipation feature are fused to obtain the local asymmetric dissipation energy flow feature .

[0113] The gray centroid normal offset feature (reflecting the offset of the gray distribution in the width direction) is greatly affected by the width scale of the filament (for example, the absolute value of the offset of a thick filament may be larger than that of a thin filament, but the relative offset degree may not be higher), and the scale difference needs to be eliminated by non-dimensionalization: the original gray centroid normal offset feature is divided by the half-width function at the center line and a very small positive number to obtain the normalized offset feature. The role of the very small positive number is to avoid the denominator being zero when the half-width function is zero (extremely thin filament), to ensure the effectiveness of the calculation; after normalization, the offset feature is no longer affected by the absolute width, and only reflects the offset degree relative to the proportion of the half-width, so that the gray offset characteristics of filaments of different widths are comparable.

[0114] The half-width dissipation feature (reflecting the change rate of the half-width along the arc length) has invalid information (non-filament characteristics), and is also affected by the width scale, which needs to be processed in two steps:

[0115] One-way constraint (reserve positive part): the nature of the filaments is to gradually thin out from the main pen, corresponding to the part of the half-width dissipation feature that decreases with the increase of arc length, that is, the positive value of the original half-width dissipation feature. If the original feature is negative, it indicates that the half-width increases, which does not conform to the characteristics of the filaments and needs to be removed. Therefore, only the positive value of the half-width dissipation feature is retained to obtain the positive dissipation part to filter the width change information of non-filament such as local thickening noise.

[0116] Non-dimensional processing: similar to the gray centroid method, the half-width function at the center line is divided by a small positive number to obtain the normalized positive dissipation feature. Thus, the influence of absolute width on the dissipation rate is eliminated, and the thinning speed (relative half-width dissipation rate) of filaments of different widths is comparable.

[0117] Edge asymmetry, normalized offset, and normalized positive dissipation are three types of features that describe the asymmetric dissipation nature of filaments from three dimensions of edge sharpness difference, gray distribution offset, and width thinning trend. The information of the three types of features needs to be integrated into a unified index through fusion: the edge asymmetry feature, the normalized offset feature, and the normalized positive dissipation feature are multiplied to obtain the local asymmetric dissipation energy flow feature. The rationality of this fusion logic lies in that all three types of features are positive indicators of filament asymmetric dissipation (edge asymmetry positive value indicates that the outside is sharper, normalized offset positive value indicates that the gray is biased towards the inside, and normalized positive dissipation positive value indicates that the width is thinning). Multiplication can realize positive superposition and negative suppression: if all three types of features are positive at a certain position, the energy flow feature value is large (strong asymmetric dissipation, confirmed as a filament); if one type of feature is negative (such as edge symmetry, width thickening), the energy flow feature value will be suppressed (weak asymmetric dissipation, excluded as a filament), and the final energy flow feature can accurately quantify the local asymmetric dissipation intensity of the filament.

[0118] In one embodiment of the present application, the local asymmetric dissipation energy flow feature is weighted and integrated along the position of the center line to obtain the filament asymmetric dissipation energy flow matrix; in the initial segment of the center line, the normal direction is locked according to the local asymmetric dissipation energy flow feature, including:

[0119] The center line The total length of the center line is obtained by trapezoidal numerical integration through the normalization parameter :

[0120] ; wherein, is the connection end of the filament and the main pen, is the filament out of the edge, ;

[0121] Define position weight , the local asymmetric dissipation energy flow feature By performing numerical integration and normalization using the trapezoidal method, the asymmetric dissipative energy flow moment of the wire is obtained:

[0122] ;

[0123] Set the initial segment ratio of the parameters Define the starting segment interval as Calculate the initial segment interval Trapezoid method numerical integration ;

[0124] Let the original normal vector be... The locked normal vector is ,but:

[0125] .

[0126] A normalized parameter is introduced for the arc length parameter of the centerline (from the main stroke connection end to the tip end). The actual total length of the centerline is obtained through numerical integration using the trapezoidal rule (since the centerline is a discretely sampled continuous curve, the trapezoidal rule can approximate the total length by summing the arc length increments of adjacent sampling points). Specifically, it is clarified that… The connection point between the connecting wire and the main pen (the starting point of the transition zone where the energy flow characteristics are most stable). Corresponding to the leading edge of the wire (the end with weaker energy flow characteristics), ensure that the start and end range of the total length is consistent with the shape of the wire.

[0127] SADF is an indicator for quantifying the global asymmetric dissipation trend of wire pulling, as detailed below:

[0128] Position weight definition: Position weights are set according to the arc length of the centerline (e.g., higher weight is given to the connecting end closer to the main stroke, or weights are allocated according to the proportion of arc length). The purpose is to integrate the local energy flow characteristics of different arc length positions according to their contribution to the overall characteristics of the silk-drawing process. For example, the energy flow characteristics of the transition area near the main stroke better reflect the essence of the silk-drawing process and should be given higher weights to avoid interference from noise at the exit point.

[0129] Trapezoidal method numerical integration: The local asymmetric dissipation energy flow characteristics are multiplied by the corresponding position weights, and then the total weighted dissipation energy is obtained by numerical integration using the trapezoidal method. The integration result reflects the weighted dissipation intensity of the entire wire drawing process.

[0130] Normalization: The weighted total dissipated energy is divided by the sum of the trapezoidal integral of the absolute value of the local energy flow characteristic and the minimum positive number to obtain the asymmetric dissipated energy flow moment (SADF) of the wire drawing process. The minimum positive number is used to avoid the denominator being zero (e.g., the energy flow characteristic of extremely fine wire drawing is close to zero), and normalization eliminates the influence of different total energy values ​​of the wire drawing process, so that the SADF can uniformly represent the global dissipation trend. The positive and negative signs reflect the dissipation direction, and the numerical value reflects the degree of concentration of dissipation.

[0131] The energy flow characteristics of the initial segment (main stroke-connecting transition zone) are the most stable and best reflect the true normal direction (the emerging end is easily affected by noise), therefore, it is necessary to focus on this segment to determine the normal direction:

[0132] Define the starting segment interval: Set the starting segment ratio (e.g., 20% of the total length). Based on the total length of the center line, define the starting segment interval from s=0 (the main pen connection end) to the arc length range of the starting segment ratio × the total length. This interval is the key area for the connection between the thread and the main pen. The asymmetry of the energy flow characteristics is most significant, which can eliminate misjudgments caused by tip noise.

[0133] Calculate the initial segment energy flow integral: Using the trapezoidal rule for numerical integration, calculate the sum of all local asymmetric dissipative energy flow characteristics within the initial segment interval (i.e., the initial segment energy flow integral). The sign of this integral result is the basis for determining whether the normal direction is correct: because the asymmetry of the initial segment energy flow characteristics is clear, the integral sign can reflect the true correspondence between the inner and outer sides.

[0134] The original normal vector may have directional ambiguity due to skeleton offset (i.e., the inner and outer labels are reversed). A unique direction needs to be locked using the sign of the initial segment's energy flow integral: Let the original normal vector be the initially defined direction, and the locked normal vector be the finally determined direction. If the initial segment's energy flow integral result is ≥0, it means the inner / outer label corresponding to the original normal vector is consistent with the actual asymmetric dissipation characteristics of the wire, requiring no adjustment; the original normal vector can be directly used. If the initial segment's energy flow integral result is <0, it means the inner / outer label of the original normal vector is reversed, and the original normal vector needs to be reversed (i.e., the locked normal vector is the opposite direction of the original normal vector).

[0135] In one embodiment of the present invention, an anisotropy metric is constructed based on edge asymmetry features and a preset magnification factor. The grayscale image is then resampled in the string-like coordinate system defined by the anisotropy metric, including:

[0136] From edge asymmetry features Extract the non-negative part ;

[0137] With preset magnification factor Line elements for constructing anisotropic measures:

[0138] ;in, Tangential along the centerline The arc length increment, To lock the normal The increment;

[0139] Set the sampling step size of the metric domain normal Euclidean normal step length in pixel domain is calculated as 0.5 pixel ;

[0140] Define the filament coordinate system Mapping function from filament coordinate to pixel coordinate:

[0141] ;

[0142] Resample the gray image to get the gray field in filament coordinate system , ; The gray value at non-integer pixel coordinate is calculated by bilinear interpolation.

[0143] The value range of edge asymmetry feature is (-1, 1), where positive value corresponds to the outside edge being sharper than the inside (asymmetry of the filament), and negative value has no practical significance (not consistent with the filament shape, may be noise or local disturbance). Therefore, only the non-negative part of the feature is extracted (i.e. when the feature value is ≥ 0, the original value is retained, and when the feature value is < 0, 0 is taken).

[0144] Line element is a parameter to describe the local scale of space, and the essence of anisotropy measurement is to adapt the scale of different directions (tangential, normal) of the filament to its asymmetric characteristics. The specific construction logic is:

[0145] Tangential (along the extension direction of the center line): the tangential arc length increment in the line element remains unchanged, i.e. the dissipation of the filament along the tangential direction is continuous, and there is no need to adjust the scale, ensuring the continuity of the features in the arc length direction;

[0146] Normal (along the width direction after locking): the normal increment in the line element needs to be multiplied by 1+ preset magnification coefficient x non-negative part of edge asymmetry feature, i.e. the larger the non-negative part of edge asymmetry feature, the sharper the outside, and the normal scale needs to be stretched through the magnification coefficient (preset fixed value, such as 0.7) to make the outside gray and gradient details clearer, and to realize the scale alignment of multi-modal features in the normal direction (avoiding the outside details being ignored due to too small scale). The final line element can adapt to the continuous tangential and asymmetric normal shape of the filament.

[0147] The measurement domain (s, r parameter domain) of the filament coordinate system needs to correspond to the pixel domain (actual image coordinates), and the normal sampling step length is as follows:

[0148] First, set a fixed normal sampling step length (0.5 pixel, to ensure uniform resampling accuracy) in the measurement domain;

[0149] ​Then, based on the anisotropy measurement, the normal step size of the metric domain is converted into the Euclidean normal step size of the pixel domain (i.e., the corresponding straight-line distance in the actual image): Since the normal scale has been adjusted by the edge asymmetry feature, the Euclidean normal step size at different positions will change with the sharpness of the outer side (the sharper the outer side, the larger the step size), ensuring that the sampling points of the metric domain can accurately cover the outer detail area in the pixel domain.

[0150] The wire-drawing coordinate system is based on the parameter domain of the centerline (s is the arc length, r is the normal coordinate), and needs to be associated with the actual pixel coordinates of the image through a mapping function. The specific rules are as follows:

[0151] For any point (s, r) in the wire-drawing coordinate system, its corresponding pixel coordinates = pixel coordinates of the center line at arc length s + normal coordinates r × locked normal vector. This function ensures that each parameter point in the wire-drawing coordinate system can find a unique physical location in the pixel domain, establishing a mapping between the parameter domain and the pixel domain.

[0152] The purpose of resampling is to normalize the grayscale information of the original pixel domain into the grid of the wire coordinate system, as follows:

[0153] Traverse all parameter points according to the grid of the wire-drawing coordinate system (s in arc length steps, r in 0.5 pixel metric domain steps);

[0154] The mapping function is used to find the pixel coordinates corresponding to each parameter point.

[0155] If the corresponding pixel coordinates are integers (falling exactly in the center of the pixel), the gray value of that pixel is directly taken; if they are non-integers (falling in the pixel gap), the gray value is obtained by bilinear interpolation (using the gray values ​​of four adjacent pixels to calculate the smooth gray value at the gap).

[0156] The set of gray values ​​for all parameter points constitutes the gray field in the wire-coordinate system. This eliminates gray-level misalignment caused by wire distortion and scale inhomogeneity in the original pixel domain, resulting in regular and smooth gray-level features.

[0157] In one embodiment of the present invention, the local asymmetric dissipative energy flow characteristics are smoothed, and the core region of the wire drawing is determined by combining the energy flow threshold determined based on the asymmetric dissipative energy flow moment of the wire drawing with the monotonic change criterion of the local asymmetric dissipative energy flow characteristics, including:

[0158] Setting parameters for energy flow smoothing scale Define a one-dimensional Gaussian kernel for 3 pixels:

[0159] ;in, is the independent variable of the Gaussian kernel function;

[0160] Characteristics of local asymmetric dissipative energy flow , and the smoothed energy flux feature is obtained by Gauss convolution along the arc length direction:

[0161] ;

[0162] Definition of the sign function of the non-symmetric dissipation energy flux moment of the thread ; wherein, , ;

[0163] Set the parameter energy flux threshold quantile to 80%, and calculate the energy flux threshold by the quantile function : The quantile function is used to return the value when the proportion of elements in the input data set that are less than or equal to the value is : The th value in :

[0164] ;

[0165] Set the arc length sampling step size to 1 pixel, and calculate the first derivative of using symmetric difference method:

[0166] Definition of the set of effective dissipation intervals :

[0167] ;

[0168] Iterate through to identify all continuous sub-intervals, and select the largest closed interval as the core interval of the thread; wherein, is the starting arc length of the core interval, and is the ending arc length of the core interval.

[0169] Local non-symmetric dissipation energy flux features may be affected by image noise (such as paper texture, scanning interference), resulting in local fluctuations, which need to be eliminated by smoothing:

[0170] First, set the energy flux smoothing scale to 3 pixels (this scale is suitable for fine structures of the thread, which can filter out small noise without damaging the overall trend of the thread energy flux); based on this scale, define a one-dimensional Gaussian kernel (the Gaussian kernel has the characteristics of high middle weight and low two-end weight, which can achieve smoothing and preserve the overall shape of the feature):

[0171] ​​The local asymmetric dissipation energy flow feature is convolved with a one-dimensional Gaussian kernel along the arc length direction of the center line (the direction of the filament extension), that is, the local fluctuations are eliminated by weighted averaging to obtain the smoothed energy flow feature. This makes the energy flow feature more consistent with the real trend of the filament gradually decaying from the main pen to the end of the stroke, avoiding misjudgment caused by noise.

[0172] The sign of the asymmetric dissipation energy flow moment (SADF) of the filament reflects the direction of the overall asymmetric dissipation of the filament (e.g., the typical direction of "sharp on the outside and blunt on the inside"), and the sign function serves as a reference for the subsequent threshold judgment:

[0173] If SADF is greater than 0, the sign function takes a value of 1; if SADF is less than or equal to 0, the sign function takes a value of -1.

[0174] The energy flow threshold is used to filter the strong asymmetric dissipation region (the energy flow feature of the filament core region is significantly higher than that of the noise region), and the logic is as follows:

[0175] Set the energy flow threshold quantile to 80% (this quantile can ensure that the filtered region is the top 20% in terms of energy flow feature intensity, corresponding to the strong dissipation characteristics of the filament core);

[0176] Calculate the 80th quantile of the absolute value of the smoothed energy flow feature in the full length interval of the center line (i.e., 80% of the values are less than this quantile and 20% of the values are greater than this quantile);

[0177] Combine the sign function to multiply the quantile value by the sign function to obtain the final energy flow threshold. This ensures that the threshold direction is consistent with the overall dissipation direction of the filament, and only filters out strong dissipation regions with the same overall trend, excluding weak dissipation noise in the opposite direction.

[0178] The first derivative is used to determine the trend of the energy flow feature, which has a monotonically decreasing characteristic from the main pen to the end of the stroke (corresponding to a non-positive derivative), which needs to be quantified by the derivative:

[0179] Set the arc length sampling step size to 1 pixel (this step size matches the discrete sampling accuracy of the center line);

[0180] Use the symmetric difference method (by subtracting the energy flow value of the adjacent previous point from the energy flow value of the adjacent next point and dividing by twice the arc length sampling step size) to calculate the first derivative of the smoothed energy flow feature along the arc length direction. This method can accurately reflect the local change rate of the energy flow feature, and a derivative less than or equal to 0 indicates that the energy flow feature is decaying, which is consistent with the trend of the filament core region; a derivative greater than 0 indicates that the energy flow is increasing, which is not consistent with the filament characteristics.

[0181] The effective dissipation interval is a region that meets both strong dissipation and monotonic decay, and needs to be filtered by both threshold and derivative conditions:

[0182] Each interval in the set of effective dissipation intervals needs to satisfy two conditions simultaneously:

[0183] The sign function x smoothed energy flux feature ≥ energy flux threshold (ensure that it is a strong dissipation region consistent with the overall trend);

[0184] The first derivative of the sign function x smoothed energy flux feature ≤ 0 (ensure that it is a monotonically decaying region, consistent with the filament energy flux trend);

[0185] All arc length segments that satisfy the above conditions form the set of effective dissipation intervals. The set of effective dissipation intervals excludes noise regions that are strongly dissipative but not monotonic, and monotonic but weakly dissipative, and only retains candidate segments that meet the filament characteristics.

[0186] There may be multiple discrete segments in the set of effective dissipation intervals (such as short strong dissipation segments caused by local noise), which need to be filtered out to obtain the most complete filament region:

[0187] Iterate through the set of effective dissipation intervals to identify all continuous subintervals (continuous arc length segments);

[0188] Select the longest continuous closed interval as the filament core interval. Since the filament is a continuous subdivision extending from the main pen, the longest continuous effective interval must be the complete filament core, avoiding misjudgment of short noise segments as filaments.

[0189] In an embodiment of the present application, a filament mask is generated in the filament coordinate system according to a preset bandwidth ratio, and energy compensation along the tangential direction is performed on the resampled grayscale image through the filament mask to obtain a standard character shape, comprising:

[0190] In the filament coordinate system , based on the filament core interval and the half-width function , the filament mask is defined :

[0191] ;

[0192] Where, is the parameter filament bandwidth ratio; is an indicator function, which takes the value 1 when the condition in the parentheses is true, and 0 otherwise;

[0193] Define a tight support window function ; where, is the normalized normal coordinate, ;

[0194] Set the parameter compensation strength based on the smoothed local asymmetric dissipation energy flux feature , the resampled grayscale image Perform energy compensation along the tangential direction:

[0195] ; wherein, is the normalized glyph in the string coordinate system;

[0196] Convert the normalized glyph in the string coordinate system Calculate the pixel position vector using the inverse mapping formula : , thereby converting to the pixel domain normalized glyph image .

[0197] The string mask is a binary template that marks the string region and needs to be circumscribed based on the string characteristics to avoid miscovering the main pen or missing the string:

[0198] In the string coordinate system (s is the arc length, r is the normal coordinate), the definition of the string mask needs to meet two conditions at the same time:

[0199] The arc length s belongs to the string core interval;

[0200] The absolute value of the normal coordinate r is ≤ preset bandwidth ratio × half-width function value (the preset bandwidth ratio is a parameter that controls the width of the string mask, such as 0.6, to ensure that the mask only covers the narrow band of the string and does not extend to the main pen).

[0201] The area that meets the above conditions is marked as 1 (indicating the string region); the area that does not meet the conditions is marked as 0 (indicating the main pen or background region). The final string mask can locate the range of the string that needs to be weakened.

[0202] Energy compensation needs to be smoothed only within the string mask to avoid distortion of the main pen shape caused by compensation traces, and the role of the tight support window function is to achieve normal smoothing transition:

[0203] First, calculate the normalized normal coordinate v: divide the normal coordinate r by the preset bandwidth ratio × half-width function value to get v, which is limited to the range [-1, 1] (only within the string mask, outside the mask, v exceeds the range, and the window function is 0);

[0204] Define a tight support window function based on v (such as a quadratic function of v, satisfying v = ±1 when the function value is 0, and v = 0 when the function value is maximum): the function continuously changes within v ∈ [-1, 1], and is 0 when v ∉ [-1, 1]. Thus, it is ensured that energy compensation is only performed within the string mask, and it is smoothly attenuated along the normal direction (the width direction of the string) from the center to the edge, avoiding the blurring of the main pen edge caused by hard cutting.

[0205] Energy compensation is to weaken the string energy rather than hard delete, ensuring that the main pen structure is complete and free of string interference, as follows:

[0206] Set compensation intensity (such as 0.3, control the degree of weakening of the pulling filament energy, avoid excessive compensation leading to main pen damage);

[0207] Based on the resampled gray image, subtract the compensation intensity x the smoothed local asymmetric dissipation energy flow feature x the compact support window function: wherein the smoothed energy flow feature reflects the dissipation intensity of each position of the pulling filament (the stronger the energy flow, the more significant the pulling filament, and more compensation is needed), and the window function ensures that the compensation only acts within the pulling filament mask and is normal smooth;

[0208] The gray value set obtained after the above operation is the standard character shape in the pulling filament coordinate system. In this character shape, the pulling filament energy attenuates along the tangential direction (the extension direction of the pulling filament) with the dissipation intensity, and the main pen energy is basically retained, achieving the effect of removing the pulling filament interference.

[0209] The standard character shape of the pulling filament coordinate system is parameter domain data, which needs to be converted to image actual pixel coordinates to be used for subsequent recognition. The conversion logic is: using the defined mapping formula of the pulling filament coordinate system to the pixel coordinate, that is, for each parameter point in the standard character shape of the pulling filament coordinate system , its corresponding coordinate in the pixel domain is calculated by the formula ; wherein is the coordinate vector of the center line at the arc length , is the unit normal vector after locking, is the normal distance. The gray value of the standard character shape at the point is assigned to the calculated pixel coordinate . The final pixel domain standard character shape image is a purified character shape with clear main pen structure and no pulling filament interference.

[0210] In an embodiment of the present application, the recognition result is obtained based on the standard character shape and the standard character shape library, including:

[0211] Input the standard character shape into the pre-trained convolutional neural network to obtain a feature map;

[0212] Calculate the similarity of the feature map and the feature map of each standard character shape in the standard character shape library, and select the standard character shape with the largest similarity as the recognition result.

[0213] The standard character shape is a purified character shape with no string interference and clear main pen structure, which is the high-quality input for recognition. The pre-trained convolutional neural network (CNN) has mature feature extraction capability of character shape structure. The specific process is as follows: the pixel domain standard character shape image is input into the CNN, and the network cooperates through multiple layers of convolution layer (capturing local stroke details such as the edge and texture of horizontal, vertical, hook and dot), pooling layer (reducing feature dimension and retaining key structure information) and fully connected layer (integrating global features to form a unified feature vector), and finally outputting a feature map (or feature vector) that can represent the structural essence of the standard character shape, that is, the feature map has removed noise and string interference, and only retains the core recognition information such as main pen layout and stroke direction.

[0214] The standard character shape library stores standard feature maps of various types of regular and cursive Chinese characters (extracted by the same pre-trained CNN from standard regular and cursive character shapes, ensuring consistent feature dimension and extraction logic), and the matching is realized through similarity calculation: the feature map output by the standard character shape is calculated with the feature map of each standard character shape in the standard character shape library (common indicators such as cosine similarity, quantifying the structural similarity of two feature maps, and the value closer to 1 indicates more consistent structure); after the traversal is completed, the standard character shape with the largest similarity is selected as the final recognition result of the Chinese character to be recognized.

[0215] The pre-training of the core is to enable the CNN to have the ability to extract the structural features of the regular and cursive character shapes, and the process needs to be carried out around the three elements of data, network and training, as follows:

[0216] Data preparation: build a regular and cursive Chinese character dataset containing a large number of annotated clear regular and cursive character shape images (covering commonly used Chinese characters, each containing multiple writing variants to improve generalization); perform data augmentation on the dataset (such as slight rotation, translation and gray scale disturbance) to avoid model overfitting, and normalize the images to a fixed size (to adapt to the network input requirements).

[0217] Network structure design: adopt a lightweight CNN structure (such as LeNet variant or simplified AlexNet), which usually contains 3-5 layers of convolution layer (with ReLU activation function to enhance non-linear feature extraction capability), 2-3 layers of pooling layer (such as maximum pooling to retain key features), 1-2 layers of fully connected layer (mapping convolution features to a fixed dimension feature vector) and 1 layer of output layer (corresponding to the number of Chinese character categories).

[0218] Model training: take the labeled stroke style image as the training input, use the cross entropy loss function (quantify the difference between the model prediction category and the real label), Adam or SGD optimizer (minimize the loss function, update the network parameters); Set a reasonable training round (such as 50-100 rounds), evaluate the model performance (such as recognition accuracy) after each round of training with the validation set, adjust the learning rate and other parameters to optimize the model; When the validation set accuracy is stable and there is no obvious overfitting, stop training.

[0219] Model saving: save the trained network parameters (convolution kernel weight, fully connected layer parameter, etc.) as a pre-trained model.

[0220] Example two

[0221] As shown in Figure 2 A Chinese character recognition system based on multi-modal feature fusion is applied to any one of the Chinese character recognition methods based on multi-modal feature fusion, comprising:

[0222] A data acquisition module acquires a gray image of a Chinese character to be recognized, extracts the skeleton of each pixel width in the gray image and the distance feature of each pixel to the nearest boundary, filters the candidate branches of the string, and fits the center line of the string candidate branch;

[0223] A feature extraction module calculates the edge asymmetry feature on the center line based on the gradient feature of the gray image; calculates the gray centroid normal offset feature on the center line based on the gray feature of the gray image; and calculates the half-width dissipation feature on the center line based on the distance feature;

[0224] A feature fusion calculation module fuses the edge asymmetry feature, the gray centroid normal offset feature and the half-width dissipation feature to obtain the local asymmetric dissipation energy flow feature on the center line; based on the position of the local asymmetric dissipation energy flow feature along the center line, the asymmetric dissipation energy flow moment of the string is obtained by weighted integration; in the initial segment of the center line, the normal direction is locked according to the local asymmetric dissipation energy flow feature;

[0225] A resampling module constructs an anisotropy metric based on the edge asymmetry feature and a preset magnification coefficient, and resamples the gray image in the string coordinate system defined by the anisotropy metric;

[0226] An interval calculation module smoothes the local asymmetric dissipation energy flow feature, and determines the string core interval by combining the energy flow threshold determined based on the asymmetric dissipation energy flow moment of the string and the monotonic variation criterion of the local asymmetric dissipation energy flow feature;

[0227] A standard stroke generation module generates a string mask in the string coordinate system according to a preset bandwidth ratio, and performs tangential energy compensation on the resampled gray image through the string mask to obtain a standard stroke.

[0228] The character recognition module obtains the recognition result based on the standard character and the standard character library.

[0229] The above describes the embodiments of the present embodiment, but the present embodiment is not limited to the above-described specific embodiments, which are only illustrative but not restrictive. Those skilled in the art can make many forms under the inspiration of the present embodiment, which are all within the protection scope of the present embodiment.

Claims

1. A Chinese character recognition method based on multi-modal feature fusion, characterized in that, Comprise: S1, obtain the gray image of the Chinese character to be identified, extract the skeleton of each pixel width in the gray image, and the distance feature of each pixel to the nearest boundary, screen the candidate branch of the string according to the skeleton, and fit the center line of the candidate branch of the string; S2, based on the gradient feature of the gray image, calculate the edge asymmetry feature on the center line; Based on the gray feature of the gray image, calculate the gray centroid normal offset feature on the center line; Based on the distance feature, calculate the half-width dissipation feature on the center line, including: At the center line where the tangent vector is computed ; wherein, denotes the arc length derivative of the center line; denotes the center line at the arc length parameter unit tangent vector; Defining a normal vector : wherein, and is the transverse and longitudinal components; Distance-based features Defining half-width function The pixel position of any point of the normal profile is ; wherein, is the normal coordinate; For a grayscale image Compute horizontal gradient using Sobel operator And vertical gradient Get spatial gradient vector ; Calculate the normal gradient derivative: ； Calculate the maximum gradient absolute value on the inside: ； Calculate the maximum gradient absolute value on the outside: ； The edge asymmetry feature is ; Defining a normal gray scale distribution ; The gray centroid normal offset feature is calculated by trapezoidal numerical integration with a sampling step of 0.5 pixels ; The half-width dissipation feature is calculated by a three-point difference method with an arc length step of 1 pixel ; wherein is an arc length difference step used when calculating the derivative Fuse the edge asymmetry feature, the gray centroid normal offset feature and the half-width dissipation feature to obtain the local asymmetric dissipation energy flow feature on the center line; Based on the position of the local asymmetric dissipation energy flow feature along the center line, weighted integration is obtained to obtain the asymmetric dissipation energy flow moment of the string; In the initial segment of the center line, the normal direction is locked according to the local asymmetric dissipation energy flow feature; S3, construct anisotropy measure based on edge asymmetry feature and preset amplification coefficient, resample gray image in string coordinate system defined by anisotropy measure; S4, smooth the local asymmetric dissipation energy flow feature, combine the energy flow threshold value determined based on the asymmetric dissipation energy flow moment of the string and the monotonic variation criterion of the local asymmetric dissipation energy flow feature to determine the core interval of the string; S5, generate string mask in string coordinate system according to preset bandwidth ratio, perform tangential energy compensation on resampled gray image through string mask to obtain standard character shape; S6, based on the standard character shape and the standard character shape library, the recognition result is obtained. 2.The Chinese character recognition method based on multi-modal feature fusion of claim 1, characterized in that, Obtain the gray image of the Chinese character to be identified, extract the skeleton of each pixel width in the gray image, and the distance feature of each pixel to the nearest boundary, screen the candidate branch of the string according to the skeleton, and fit the center line of the candidate branch of the string, including: acquiring a gray-scale image of a Chinese character to be recognized ; If the grayscale image has the th If the grayscale value of the nth pixel is less than or equal to the grayscale threshold, then the nth pixel will be... Each pixel is marked as 1, otherwise it is marked as 0, to obtain a binary image. ; Determining a set of foreground boundary pixels of a binary image ; wherein each foreground boundary pixel has a value of 1 and each non-boundary pixel has a value of 0 and there exists a pixel within the 8-neighborhood of each foreground boundary pixel such that ; Get the The Euclidean distance between the coordinates of the nth pixel and the coordinates of each foreground boundary pixel is calculated, and the minimum Euclidean distance is selected as the nth pixel. Distance features per pixel ; Defining a skeleton set ; where the skeleton set has two skeleton pixels and satisfy: such that: ,in, Represents the first element in the skeleton set. One skeleton pixel, Indicates the first Distance features of each skeleton pixel, It is the Euclidean norm; The first skeleton set The pixel width of each skeleton pixel is ; Collecting skeletons The maximum pixel width of all skeleton pixels in , is denoted as As the main pen width threshold , and define the main pen connected region ; In the connected graph of the skeleton set , the starting point of screening belongs to the main pen connected region The path of As a candidate branch of the string; wherein, The arc length parameter of the path, The total arc length parameter of the path satisfies: path length ; wherein, is a first derivative of a path length, is a preset minimum length threshold; maximum pixel width of all pixels on the path except the start point ; wherein is the maximum width upper limit; Derivative of path width along arc length Holds for more than 70% of the total path length; where, is the monotonic thinning rate threshold, the derivative is calculated by a three-point difference method, i.e., , is the arc length sampling step; Collect candidate branches of the silk thread Fitted pixel set ;in, To fit the number of skeleton pixels in the pixel set, To fit the set of pixels, the first The pixel coordinates of each skeleton pixel; performing a cubic spline fit on the set of skeleton pixels by minimizing an energy function obtaining a centerline where the energy function is: ； wherein, is a skeleton pixel in the set of pixels corresponding arc length parameter, is a centerline second derivative of, is a smoothing weight. 3.The Chinese character recognition method based on multi-modal feature fusion of claim 2, characterized in that, Fuse the edge asymmetry feature, the gray centroid normal offset feature and the half-width dissipation feature to obtain the local asymmetric dissipation energy flow feature on the center line, including: Gray centroid method to offset features Performing dimensionless processing, get normalized offset features ; wherein, Is a minimum positive number; Half-width dissipation characteristics One-way constraint and dimensionless processing are performed, including: Retaining half-width dissipation of forward portion ; Computing normalized forward dissipation features ; Fusion edge asymmetry feature, normalized offset feature and normalized positive dissipation feature, get local asymmetric dissipation energy flow feature . 4.The Chinese character recognition method based on multi-modal feature fusion of claim 3, characterized in that, Based on the position of the local asymmetric dissipation energy flow feature along the center line, weighted integration is obtained to obtain the asymmetric dissipation energy flow moment of the string; In the initial segment of the center line, the normal direction is locked according to the local asymmetric dissipation energy flow feature, including: centerline the total length of the centerline by trapezoidal numerical integration with normalization parameters : ; wherein, is the connection end of the string to the main pen, is the string exit end, ; Defining position weight characteristic of local asymmetric dissipation energy flow Lagrange numerical integration and normalization are performed to obtain the filament asymmetric dissipation energy flow moment: ； Setting parameter initial segment ratio , defining initial segment interval as , calculating trapezoidal numerical integration value of in initial segment interval ; Let the original normal vector be , the normal vector after locking be , then: 。 5.The Chinese character recognition method based on multi-modal feature fusion of claim 4, characterized in that, Construct anisotropy measure based on edge asymmetry feature and preset amplification coefficient, resample gray image in string coordinate system defined by anisotropy measure, including: Extracting non-negative portions from edge asymmetry features ;​ with a preset amplification factor Constructing line elements of anisotropy metrics: ; wherein, is the arc length increment along the centerline tangent to the centerline, is the increment along the locking normal to the locking. Setting metric domain normal sampling step size 0.5 pixels, calculate the corresponding pixel domain euclidean normal step size ; Defining the wire coordinate system Mapping function to pixel coordinates: ； for a grayscale image resampling to obtain a grayscale field in the wire coordinate system , ; wherein the grayscale value at the non-integer pixel coordinate is calculated by bilinear interpolation. 6.The method of claim 5, wherein, Smooth the local asymmetric dissipation energy flow feature, combine the energy flow threshold value determined based on the asymmetric dissipation energy flow moment of the string and the monotonic variation criterion of the local asymmetric dissipation energy flow feature to determine the core interval of the string, including: Setting parameter energy flow smoothing scale For 3 pixels, define a one-dimensional Gaussian kernel: ; wherein, is an argument of a Gaussian kernel function; Local asymmetric dissipation energy flow feature Gaussian convolution is performed in the arc length direction to obtain a smoothed energy flow feature: ； Sign function defining the wire-drawing asymmetric dissipation energy flow moment ; wherein when , when ; Setting parameters: energy flow threshold quantiles It is 80%, through the quantile function quantile function This is used to return the proportion of elements in the input data set that are less than or equal to this value. The value at time, calculation Inside The The position value is used to determine the energy flow threshold: ； Setting arc length sampling step For 1 pixel, calculate the first derivative using symmetric difference method ;​ Defining an effective dissipation interval set : ； traverse identify all consecutive sub-intervals, select the largest length closed interval as the core interval of the string; wherein, the core interval start arc length, the core interval end arc length. 7.The method of claim 6, wherein, Generate string mask in string coordinate system according to preset bandwidth ratio, perform tangential energy compensation on resampled gray image through string mask to obtain standard character shape, including: In the wire coordinate system , based on the wire core interval and half-width function define the wire mask : ； wherein is a parameter string tension bandwidth ratio; is an indicator function that takes the value 1 when the condition in the parentheses is true, and 0 otherwise. Defining a compactly supported window function ; wherein is a normalized normal coordinate, ; Setting parameter compensating intensity based on smoothed local asymmetric dissipated energy flow features on the resampled grayscale image performing tangential energy compensation: ; wherein is the normalized glyph in the wire coordinate system; converting the glyph image into the pixel domain calculating the pixel position vector using the inverse mapping formula : , thereby converting the glyph image into the pixel domain . 8.The Chinese character recognition method based on multi-modal feature fusion of claim 7, characterized in that, Based on the standard character shape and the standard character shape library, the recognition result is obtained, including: Input the standard character shape into the pre-trained convolutional neural network to obtain the feature map; Calculate the similarity between the feature map and the feature map of each standard character shape in the standard character shape library, and select the standard character shape with the maximum similarity as the recognition result.

9. A Chinese character recognition system based on multi-modal feature fusion, applied to the Chinese character recognition method based on multi-modal feature fusion in any one of claims 1-8, characterized in that, Comprise: The data acquisition module acquires a gray image of the Chinese character to be recognized, extracts a skeleton of each pixel width in the gray image and a distance feature of each pixel to the nearest boundary, screens a string candidate branch according to the skeleton, and fits a center line of the string candidate branch; The feature extraction module calculates an edge asymmetry feature on the center line based on a gradient feature of the gray image; The feature extraction module calculates a gray centroid normal offset feature on the center line based on a gray feature of the gray image; The feature extraction module calculates a half-width dissipation feature on the center line based on the distance feature, including: At the center line where the tangent vector is computed ; wherein denotes the arc length derivative of the centerline; denotes the centerline at the arc length parameter unit tangent vector; Defining a normal vector : wherein and are the transverse and longitudinal components, respectively; Distance-based features Defining half-width function The pixel position of any point of the normal profile is ; wherein, is the normal coordinate; For a grayscale image Compute horizontal gradient using Sobel operator And vertical gradient Get spatial gradient vector ; calculating a normal gradient derivative; ； calculating an inside maximum gradient absolute value; ； calculating an outside maximum gradient absolute value; ； The edge asymmetry feature is ; Defining a normal gray scale distribution ; The gray centroid normal offset feature is calculated by trapezoidal numerical integration with a sampling step of 0.5 pixels ; The half-width dispersion feature is calculated by a three-point difference method with an arc length step of 1 pixel ; wherein is an arc length difference step used when calculating the derivative; The feature fusion calculation module fuses the edge asymmetry feature, the gray centroid normal offset feature and the half-width dissipation feature to obtain a local non-symmetry dissipation energy flow feature on the center line, integrates the local non-symmetry dissipation energy flow feature along the position of the center line based on a weighting, and obtains a string non-symmetry dissipation energy flow moment; in a starting segment of the center line, the normal direction is locked according to the local non-symmetry dissipation energy flow feature; The resampling module constructs an anisotropy metric based on the edge asymmetry feature and a preset magnification coefficient, and resamples the gray image in a string coordinate system defined by the anisotropy metric; The interval calculation module smoothes the local non-symmetry dissipation energy flow feature, combines an energy flow threshold value determined based on the string non-symmetry dissipation energy flow moment and a monotonic variation criterion of the local non-symmetry dissipation energy flow feature, and determines a string core interval; The standard character shape generation module generates a string mask in the string coordinate system according to a preset bandwidth ratio, performs tangential energy compensation on the resampled gray image through the string mask, and obtains a standard character shape; The character shape recognition module obtains a recognition result based on the standard character shape and a standard character shape library.

Images