Bent and rotating text detection and recognition method and storage medium

Abstract

The invention discloses a bent and rotated text detection and recognition method, and relates to the technical field of computer vision. The method comprises the steps that firstly, an input image is processed through a multi-view fusion detection mechanism, and an optimal detection view image is generated and screened out to eliminate large-angle rotation interference; secondly, on the basis of the optimal view angle image, parameterized geometric modeling is carried out on a text bending boundary by adopting a segmented cubic Hermite interpolation curve, and a smooth and continuous text contour curve is constructed; then, a Hermite curve-based feature alignment unit is utilized to carry out adaptive sampling and mapping on image features according to the contour curve, and a curved text region is converted into a regular rectangular feature map so as to eliminate geometric deformation; and finally, inputting the rule feature map into the text recognition model, and outputting a recognition result. According to the method, the detection and recognition precision and robustness of the bent and rotating text in a complex industrial scene are greatly improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a method for detecting and recognizing curved and rotated text based on Hermite curves and a multi-kernel fusion mechanism. Background Technology

[0002] Scene text detection, as a core task of OCR technology, aims to accurately locate text regions from a target natural scene, providing a foundation for subsequent text recognition tasks. With continuous technological advancements, existing text detection methods have achieved good results in some simple scenarios. However, in complex industrial scenarios, text often faces problems such as large-angle curvature and extreme geometric deformation. Existing methods are generally unsuitable for such applications due to limitations in rotational robustness and inaccurate geometric modeling.

[0003] Existing methods for text detection with large-angle curvature and rotation mostly have significant drawbacks. Regression-based methods typically predict the contour control points and curve parameters of the text directly. The interpolation between control points relies on the polynomial curve assumption, resulting in weak edge constraints on the control points. When curvature changes abruptly, they do not explicitly model the relationship between the curvature direction and rotation angle, potentially leading to poor fitting ability for text with large-angle curvature and rotation. Segmentation-based methods rely on pixel-level classification, predicting text position by generating text masks. Compared to regression methods, they can better represent text with large-angle curvature, but they lack robustness to rotation scenarios, easily leading to poor separation of adjacent text. Furthermore, the complexity of post-processing results in low inference speed. Summary of the Invention

[0004] In view of the technical defects and drawbacks existing in the prior art, embodiments of the present invention provide a method and storage medium for detecting and recognizing curved and rotated text to overcome the above problems or at least partially solve the above problems. The specific solution is as follows:

[0005] As a first aspect of the present invention, a method for detecting and recognizing curved and rotated text is provided, the method comprising the following steps performed sequentially:

[0006] S1. Through a multi-view fusion detection mechanism, the input image containing curved or rotated text is analyzed and processed to generate and filter the optimal detection view image for the text region, so as to eliminate the detection interference caused by large-angle rotation.

[0007] S2. Based on the optimal detection viewpoint image, a piecewise cubic Hermite interpolation curve is used to perform parametric geometric modeling on the curved boundary of the text, thereby constructing a smooth and continuous text contour curve.

[0008] S3. Using a feature alignment unit based on Hermite curves, the image features are adaptively sampled and mapped according to the text contour curve to convert the curved text region into a regularized rectangular feature map to eliminate geometric deformation.

[0009] S4. Input the regularized rectangular feature map into the text recognition model and output the final text recognition result.

[0010] This invention effectively overcomes the detection challenge of large-angle rotated text through multi-view fusion and scoring mechanisms, ensuring the integrity and accuracy of the initial detection results. By employing piecewise cubic Hermite interpolation curves, it can accurately fit the complex contours of arbitrarily curved text, providing an accurate geometric basis for subsequent correction. Based on the precise geometric model, feature sampling and mapping are performed to achieve lossless transformation of curved text into regular rectangles, preserving the original feature information to the greatest extent and significantly improving the performance of the recognition model.

[0011] This invention provides a complete solution from detection to recognition, which is particularly suitable for industrial vision scenarios with extremely high requirements for accuracy and robustness.

[0012] Furthermore, the multi-view fusion detection mechanism in step S1 specifically includes the following steps:

[0013] S11. Process the input image using an instance segmentation model to generate a mask for the target object region, and extract the minimum bounding rectangle region I containing the target object based on this mask. crop ;

[0014] S12. Based on the rotational symmetry of the target object, divide the rectangular region I... crop Dividing along its horizontal and vertical midlines, four sub-regions {R} are obtained, each covering a different aspect of the target object. top ,R bottom ,R left ,R right};

[0015] S13. Apply a rotation transformation to each of the sub-regions to make them consistent with the preset direction, so as to eliminate the text direction diversity caused by the rotation of the target object itself, and stitch the transformed sub-regions in pairs to form a stitched image with multiple viewpoint normalization.

[0016] S14. Input the multiple stitched images into the text detection model in parallel to obtain the detection results of the text in each stitched image;

[0017] S15. Transform the detection results of each stitched image back to the original coordinate system, and then perform a comprehensive scoring function S based on the preset coordinate system. finalThe confidence and completeness of text detection results from the perspective of each sub-region are evaluated, and the sub-region with the highest comprehensive score is finally selected as the optimal detection perspective.

[0018] This mechanism generates stitched images from multiple normalized perspectives through symmetrical partitioning and rotational transformation, enabling parallel detection and ensuring that there is always a perspective that can present the target in a form that most closely resembles the regular text, thus greatly improving the success rate of initial detection.

[0019] This invention does not simply merge all detection results, but evaluates the results after inverse transformation through a comprehensive scoring function, selects the result from the optimal perspective, effectively avoids the conflicts and errors that may be caused by the fusion of multiple results, and ensures the reliability and accuracy of the final output result.

[0020] Furthermore, the implementation of step S11 specifically includes the following steps:

[0021] S11a, Model Application: The SAM (Segment Anything Model) instance segmentation model based on deep learning is used to process the input complex industrial scene image containing curved and rotated text.

[0022] S11b, Mask Generation: Using the adaptive prompting mechanism of the SAM model, an accurate binary mask M for the target object region is automatically generated;

[0023] S11c, Region Extraction: Based on the binary mask M, calculate its minimum bounding rectangle, and crop this rectangular region from the original input image to obtain the image region I to be processed, which contains only the target object. crop This provides input for subsequent symmetric subregion partitioning.

[0024] This invention employs an advanced SAM model that can more accurately generate masks for target objects, and can accurately separate text regions even in complex industrial backgrounds (I crop This provides a clean and accurate input for subsequent symmetrical partitioning, avoiding interference from background noise.

[0025] Furthermore, the comprehensive scoring function in step S15 is used to evaluate the complete coverage of the text by a single sub-region. Its function definition is based on detection confidence and spatial integrity, and its expression is:

[0026]

[0027] Where: c is the average confidence level of the detection box, calculated using the formula: c long For the confidence level of the outer long text box, c short1 and c short2d represents the confidence level of the two short text boxes; d is the Euclidean distance from the center point of the outer long text box to the center of the target object; r is the radius of the target object; α, β, λ are preset weight parameters.

[0028] This invention designs a comprehensive scoring function that combines detection confidence and spatial geometric information, enabling quantitative evaluation of the quality of detection results for different sub-regions. It not only considers the confidence level (c) of the detection model itself but also introduces a geometric consistency measure (exp(-λ·d / r)) of the relative position of the detection box and the center of the target object, making the selection criteria more comprehensive and scientific, superior to methods relying solely on confidence levels.

[0029] Furthermore, when the polar angular distance between the center of the detection box and the boundary of the sub-region is less than the threshold Δθ thr When this happens, the text is determined to be segmented text that spans adjacent sub-regions;

[0030] At this time, the comprehensive scoring function S final A third text integrity weight term needs to be introduced, and its function expression is expanded as follows:

[0031] ;

[0032] Where: L local L is the arc length of the outer contour of the detection box within the current sub-region, calculated using the piecewise cubic Hermite interpolation curve; global The estimated total arc length of the global outer contour of the detection box is given by the value of the arc length L of the segmented text in each adjacent sub-region. local The sum of; γ is the preset integrity weighting coefficient.

[0033] This invention introduces a third arc length proportional term (γ·L) local / L global It can assess the completeness of the text contained in the current sub-region, and thus tend to select the sub-region perspective that can retain more complete text information, ensuring the integrity of text information and avoiding misidentification.

[0034] Furthermore, in step S2, piecewise cubic Hermite interpolation curves are used to perform parametric geometric modeling of the text boundary. The specific implementation includes the following steps:

[0035] S21. Curve Parametric Definition: Represents the geometry of the text boundary as a coordinate function of an independent parameter t, for a sequence of nodes. Divided intervals Where i = 0, 1, ..., n-1, a local parameter s is defined to control the interpolation position of a single curve segment:

[0036]

[0037] in The interval length;

[0038] S22. Basis Function Construction: The function values ​​and first derivative values ​​at the nodes are combined using the cubic Hermite basis functions defined below to ensure the continuity of the first derivative of the curve at the nodes:

[0039]

[0040] in, The starting function value weight, The starting derivative value is the weight. Weights for the endpoint function value. The final derivative value is the weight;

[0041] S23. Generation of curve segment equations: Based on the aforementioned basis functions, in the interval... Within the curve, the parametric equations of the coordinate components x(t) and y(t) are determined by the endpoint coordinates and their derivative values, and are expressed as follows:

[0042] ;

[0043] Where, m xi , m yi For node P i =(x i ,y i The first parametric derivative at ().

[0044] S24. Overall curve synthesis: Combine the curves of each interval segment P(t)=(x(t),y(t)) T Connect the parameters t∈[ti,ti+1] in order to form a complete smooth text boundary curve.

[0045] The piecewise cubic Hermite interpolation of this invention can construct a smooth and continuous (C1 continuous) curve by utilizing the function values ​​and derivative values ​​at the nodes, which can fit the complex boundaries of curved text with extremely high accuracy and provide high-quality geometric prior information for subsequent feature alignment.

[0046] Furthermore, in step S23, the derivative of the node parameter m xi , m yi The calculation method is as follows:

[0047] S231. Calculation of derivatives for internal nodes: For internal nodes P that are not endpoints... i (i=1,2,…,n-1), to ensure that the tangent directions of adjacent curve segments are consistent at the nodes, their derivatives are calculated using the central difference method, and the formula is:

[0048]

[0049] S232. Calculation of derivatives at endpoint nodes: For the starting node P0 and the ending node Pn, their derivatives are calculated using the one-sided difference method, and the formula is as follows:

[0050]

[0051]

[0052] S233. Derivative continuity guarantee: By assigning a uniform derivative value to the internal nodes through the central difference method, the first derivative of the curve at the internal nodes is ensured to be continuous, thereby guaranteeing the smoothness of the overall contour curve.

[0053] This invention calculates the derivatives of internal nodes using the central difference method, ensuring that the entire curve is not only positionally continuous at the connection points, but also tangent in direction (continuous first derivative). This eliminates "sharp points" or abrupt turns on the model contour, making the generated text contour more natural and accurate, which is crucial for subsequent feature alignment.

[0054] Furthermore, the Hermite curve-based feature alignment unit in step S3 performs a lossless mapping from the curved text region to the regular feature space according to the following steps:

[0055] S31. Calculation of Outer Curve Arc Length and Generation of Sampling Point Parameters: For the outer curve of the text outline, calculate the arc length of each segment interval [t]. i ,t i+1 Perform numerical integration to calculate its actual arc length L. outer,i :

[0056]

[0057] And divide the arc length evenly into , generate normalized parameter sequence s k =k / L outer,i where k=0,1,…,L outer,i s0=0 corresponds to the starting point of the segment, s Louter,i =1 corresponds to the end point of the segment;

[0058] S32. Synchronous calculation of inner and outer curve node coordinates: using piecewise cubic Hermite interpolation matrix. The coordinates of the inner and outer curve nodes are obtained by solving the matrix expression of the piecewise cubic Hermite interpolation curve. Expanded as follows:

[0059]

[0060] The inner curve is forced to use the same number of segments and parameter sequence s as the outer curve. k The inner and outer curves are calculated using the matrix expression of the piecewise cubic Hermite interpolation curves with respect to parameter s. k The coordinates of the interpolation point at:

[0061]

[0062]

[0063] Where P is the path point coordinate matrix, and each row corresponds to a parameter. The interpolation point coordinate vector at ( , B is a basis function matrix, with each row corresponding to a parameter value. The elements consist of cubic Hermite basis functions and the scaling term h of the derivative, meaning each row of elements consists of basis functions h. 00 (s k ), h 01 (s k ), h h 10 (s k ), h h 11 (s k The matrix is ​​composed of ( ), where C is the control parameter matrix and its row vectors are the corresponding intervals. The coordinates of the endpoints and their derivative (x) i ,y i ,m xi ,m yi ) T ;

[0064] S33. Text height calculation and normal direction sampling parameter generation: For each pair of inner and outer circle nodes (P outer (s k ),P inner (s k )) Calculate the text height H at this location. i The average of the Euclidean distances between the first and last nodes:

[0065]

[0066] Divide the height direction evenly into H i Generate height-oriented normalized parameters. v0=0 corresponds to the point on the outer curve, v Hi =1 corresponds to a point on the inner curve;

[0067] S34. Orthogonal sampling grid construction: For each parameter pair (s k ,v jInterpolation is performed along the direction connecting the inner and outer ring nodes to calculate the coordinates of the sampling point Q:

[0068]

[0069] S35. Bilinear interpolation feature extraction and regular grid filling: For each sampling point Q(k,j), the coordinates (x,j) of the grid are... k,j ,y k,j Feature values ​​are extracted from the original feature map F using bilinear interpolation. And fill it into the corresponding positions of the regular rectangular grid. Where Δw and Δh are preset pixel intervals; finally, a regular feature map of size l×w is generated, where l=(L outer +1) Δ w , w=(H i +1) Δh.

[0070] This invention ensures that each pixel in the regular feature map precisely corresponds to a certain position in the text region of the original image by uniformly sampling (s direction) and uniformly sampling along the height direction (v direction) by calculating the actual arc length. The sampling density is adaptive to the curvature and size of the text, thus guaranteeing the integrity and alignment accuracy of the feature information.

[0071] Furthermore, the bilinear interpolation method in step S35 is performed according to the following specific steps to achieve a high-fidelity feature mapping from the original feature map to the regular grid:

[0072] S351. Nearest Neighbor Integer Coordinate Search: For any sampling point Q(k,j) with floating-point coordinates (xk,j,yk,j), first find its four nearest integer coordinate pixels. , , , ,in , , Indicates the floor operation:

[0073] S352. Decimal Offset Calculation: Calculate the decimal offset (Δx, Δy) between the sampling point coordinates and the reference integer coordinates (a, b). The calculation formula is as follows:

[0074] ,

[0075] S353. Bilinear weight calculation: Based on the decimal offset (Δx, Δy), calculate the bilinear interpolation weights corresponding to the four neighboring pixels. The calculation formula is as follows:

[0076]

[0077]

[0078]

[0079]

[0080] S354. Feature Value Fusion: Based on the calculated weights, the feature values ​​of the four neighboring pixels are weighted and summed to obtain the final interpolated feature value F at the sampling point Q(k,j). align(k,j) :

[0081] = + + +

[0082] in, This represents the feature value at the corresponding coordinate on the original feature map.

[0083] In the final step of feature alignment, this invention uses bilinear interpolation to extract feature values ​​of sampling points from the original feature map. Compared with nearest neighbor interpolation, this method can effectively reduce jagged artifacts caused by coordinate floating-point conversion, generating smoother and higher-quality regular feature maps. This provides better input for subsequent recognition networks and helps improve recognition accuracy.

[0084] As a second aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a computer, the computer program causes the computer to perform the steps of any of the above-described methods for detecting and recognizing curved and rotated text.

[0085] The present invention has the following beneficial effects:

[0086] This invention proposes an end-to-end framework that overcomes the aforementioned challenges through the collaborative optimization of geometry-driven and feature learning. The invention uses piecewise cubic Hermite curves to geometrically model text curves, setting basis functions at nodes to ensure the continuity of their first derivatives, thereby guaranteeing the generation of high-precision and smooth contour curves. The invention proposes a HermiteAlign layer, based on orthogonal sampling grids and bilinear interpolation of Hermite curves, to achieve lossless mapping of curved text to a regular feature space, significantly reducing feature distortion rate and eliminating the coupling effect between geometric sensitivity and feature distortion. Furthermore, this invention proposes a multi-kernel bounding box fusion mechanism that utilizes the rotational symmetry of industrial scenes to segment the image into multi-directional sub-regions. Multiple angle perspectives are obtained through rotation transformation, and a semantic integrity evaluation function is used for dynamic filtering to select the optimal detection perspective for text with large-angle rotation. Attached Figure Description

[0087] Figure 1 This is a flowchart illustrating the curved and rotated text detection and recognition method provided in an embodiment of the present invention. Detailed Implementation

[0088] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0089] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0090] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0091] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0092] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0093] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0094] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0095] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a method for detecting and recognizing curved and rotated text. Figure 1 This is a flowchart illustrating a method for detecting and recognizing curved and rotated text according to an embodiment of the present invention. The method includes the following steps performed sequentially:

[0096] S1. Viewpoint Optimization Processing: Through a multi-viewpoint fusion detection mechanism, the input image containing curved or rotated text is analyzed and processed to generate and filter the optimal detection viewpoint image for the text region, so as to eliminate detection interference caused by large-angle rotation.

[0097] S2. Hermite Curve Modeling: Based on the optimal detection viewpoint image, a piecewise cubic Hermite interpolation curve is used to perform parametric geometric modeling of the curved boundary of the text, thereby constructing a smooth and continuous text contour curve.

[0098] S3. Hermite curve feature alignment: Using the Hermite curve-based feature alignment unit, the image features are adaptively sampled and mapped according to the text contour curve, and the curved text region is converted into a regular rectangular feature map to eliminate geometric deformation.

[0099] S4. Recognition result output: Input the regularized rectangular feature map into the text recognition model and output the final text recognition result.

[0100] This invention proposes an end-to-end framework that overcomes the aforementioned challenges through the collaborative optimization of geometry-driven and feature learning. The invention uses piecewise cubic Hermite curves to geometrically model text curves, setting basis functions at nodes to ensure the continuity of their first derivatives, thereby guaranteeing the generation of high-precision and smooth contour curves. The invention proposes a HermiteAlign layer, based on orthogonal sampling grids and bilinear interpolation of Hermite curves, to achieve lossless mapping of curved text to a regular feature space, significantly reducing feature distortion rate and eliminating the coupling effect between geometric sensitivity and feature distortion. Furthermore, this invention proposes a multi-kernel bounding box fusion mechanism that utilizes the rotational symmetry of industrial scenes to segment the image into multi-directional sub-regions. Multiple angle perspectives are obtained through rotation transformation, and a semantic integrity evaluation function is used for dynamic filtering to select the optimal detection perspective for text with large-angle rotation.

[0101] In some embodiments, the viewpoint optimization process specifically includes:

[0102] The core idea of ​​multi-core bounding boxes is to utilize the geometric symmetry of target objects in industrial scenes to segment the original image into multi-directional sub-regions, and select the optimal detection perspective through transformation and evaluation, thereby overcoming the limitations of traditional methods in text detection with large-angle rotation.

[0103] First, input the original image I. Use SAM to segment the circular planar region of image I. Generate a target mask M through an adaptive cueing mechanism and extract the minimum bounding square region. Ensure the steel coil is completely contained within the square. Because the surface of the steel coil is a circular plane, its text distribution exhibits rotational symmetry. [The image is then displayed as a square.] The image was divided along both the horizontal and vertical midlines, resulting in four equal-sized sub-regions, each containing different portions of the target region. Divide into upper half regions along the horizontal midline With the lower half of the region Divide the region into the left half along the vertical midline. With the right half of the region The four regions correspond to the four semicircular directions of the steel coil. It is easy to see that the four semicircles together cover the entire circumference, and any point is simultaneously covered by two semicircles.

[0104] According to the pigeonhole principle, any continuous text segment on a steel coil with a length not exceeding π / 2 will be completely covered by a semicircle. Therefore, it is only necessary to find the image with the highest text integrity for recognition.

[0105] Therefore, each sub-region is rotated to align with a preset direction, thus eliminating detection interference caused by the diversity of text orientations. To improve efficiency, the rotated sub-regions are paired to form two squares, which are then used for batch 2 detection. The detection results are then mapped back to the original semicircle.

[0106] For the output detection box coordinates, perform an inverse rotation transformation based on the corresponding angle of its respective stitching sub-region. Then, convert the inversely rotated local coordinates into global polar coordinates of the original steel coil image and calculate the polar angle of the detection box. And then according to Determine the original semicircular region to which it belongs.

[0107] To compare the text integrity in candidate images and select the optimal image, this paper employs a strategy that fuses detection confidence and spatial integrity. The comprehensive scoring function for candidate sub-images is defined as follows:

[0108] ,

[0109] in, The average confidence score of the three text boxes reflects the reliability of the detection. This is the distance from the center of the outer long text to the center of the steel roll circle. For the radius of the steel coil, the exponent term is... Quantify spatial integrity.

[0110] To address the scenario where a text is segmented into multiple sub-regions, the following processing mechanism has been added: if the center angle of the detection box... The nearest semicircle boundary line, and the distance from the boundary is less than the threshold. If the text is segmented, the adjacent semicircles covered by the text are recorded. The outer arc length of the detection box in each semicircle is calculated using Hermite curves. ,choose The larger semicircle is used as the main region. The scoring function is then defined as follows:

[0111] ,

[0112] in Let be the arc length of the detection box in the current semicircle. To estimate the global arc length (i.e., the sum of the arc lengths of the two parts), This is the integrity weight.

[0113] Prioritize the semicircle with high integrity; if a certain semicircle... If , then select directly without needing to complete; if there is no complete coverage, then select... The highest semicircle is used as the main region. Missing parts are filled in using Hermite interpolation, utilizing the endpoint of the main region and the starting points of adjacent regions. Finally, the selected optimal image is rotated back to the original coordinate system of the steel coil.

[0114] In some embodiments, Hermite curve modeling includes:

[0115] Piecewise cubic Hermite interpolation curves are used for parametric modeling of text boundaries. Piecewise cubic Hermite curves only require labeling the endpoints of each segment, and the curve shape is controlled by the derivative. At the nodes, the Hermite curve ensures the continuity of its first derivative through the setting of basis functions, meaning that the tangent direction of adjacent curve segments is consistent at the connection point, thus ensuring the smoothness of the overall curve and avoiding the problem of curve polylines. For text with large-angle curvature, only 6 endpoints need to be labeled on each of the top and bottom edges to generate a high-precision smooth curve, significantly reducing labeling costs.

[0116] Piecewise cubic Hermite interpolation curves construct a smooth curve that passes through a given control point and satisfies derivative constraints by representing each coordinate component (x(t), y(t)) in a plane or space as a cubic polynomial function of a certain independent parameter t.

[0117] First, a set of nodes is given. Corresponding coordinate points Set global parameters , The range of values ​​is , is used to represent the progress (arc length) of a curve. Within an interval Within a certain segment, define a local parameter s to control the interpolation position along the curve segment:

[0118]

[0119] in The interval length is, i.e. Time corresponding to the starting point , Time corresponds to the end point .

[0120] Basis functions are used to combine function values ​​and derivatives at nodes to generate cubic polynomials. Normalization parameters... The basis functions are:

[0121]

[0122] in, The starting function value weight, The starting derivative value is the weight. Weights for the endpoint function value. The final derivative value is the weight. When hour =1, the starting function value weight is 1, corresponding to the starting point hour =1, the weight of the endpoint function value is 1, corresponding to the endpoint In the interval Inside, and The expression is:

[0123] ;

[0124] in and is the derivative of the parameter at the node, and h is used to scale the derivative value to match the rate of change of the parameter s.

[0125] When calculating the derivative values ​​of control points, the central difference method is used for interior points:

[0126]

[0127] One-sided difference method is used for the endpoints:

[0128]

[0129]

[0130] Note that there are two consecutive intervals. and The endpoints are respectively , The endpoint of the previous interval and the starting point of the next interval are both... Because in the Hermite curve, the derivative values ​​of the internal nodes are not calculated independently by adjacent curve segments, but are solved by a unified central difference formula. This mechanism ensures that the derivative values ​​at the connection points of two continuous intervals are strictly equal, thereby ensuring the continuity of the first derivative of the curve and making the curve smooth.

[0131] Ultimately and Combining these, we obtain the parametric equations for the piecewise cubic Hermite interpolation curve:

[0132]

[0133] By observing directed or curved scene text in existing public datasets and real-world examples, and through experiments demonstrating that piecewise cubic Hermite interpolation curves are sufficient to fit various types of directed or curved scene text, we can see that this method is effective. In this approach, a piecewise cubic Hermite curve is used for each of the upper and lower boundaries of the text, with each curve labeled with 6 endpoints, resulting in a total of 12 control points. Therefore, the detection branch only requires a single convolutional layer with 24 output channels to learn. and Positioning text of any shape can be accomplished by offsetting the x and y values ​​of each control point.

[0134] In some embodiments, feature alignment of Hermite curves specifically includes:

[0135] After detecting the Hermite curve, the HermiteAlign operation aligns the features of the curved text region into a regular grid, facilitating subsequent text recognition. The sampling grid of HermiteAlign is not rectangular; instead, it dynamically generates sampling points orthogonal to the text boundary along the Hermite curve. The sampling points maintain equal spacing in both width and height, and are bilinearly interpolated relative to the coordinates. Simultaneously, the continuity of the first derivative of the Hermite curve ensures that the sampling grid is uniformly distributed along the text boundary, avoiding feature distortion. The specific operation steps are as follows:

[0136] In the interval Inside, and The expression is:

[0137]

[0138] in .

[0139] The curved text is fitted by two curves, an inner circle and an outer circle, and finally straightened into a rectangle. Therefore, it is necessary to ensure that the number of segments and nodes of the inner and outer circle curves are strictly aligned. Although the lengths of the inner and outer circle arcs are not equal, they are equal when mapped onto opposite sides of the rectangle. Therefore, we only need to consider the outer circle operation, and force the same parameters to be applied to the inner circle.

[0140] For each outer curve interval First, its actual arc length is calculated through numerical integration. :

[0141]

[0142] And divide the arc length evenly into Parts, generated Each node corresponds to a normalized parameter. ( ), =0 indicates the starting point of this curve segment. =1 indicates the endpoint of this curve segment. To maintain alignment between the inner and outer curve nodes, corresponding segments of the inner curve are forced to use the same number of nodes. Ensure parameters Consistent.

[0143] Then, the piecewise cubic Hermite interpolation matrix is ​​used. The coordinates of the inner and outer curve nodes are obtained by solving the matrix expression of the piecewise cubic Hermite interpolation curve. Expanded as follows:

[0144] ;

[0145] Where P is the path point matrix, and each row corresponds to a parameter. The interpolation point coordinates at ( , B is a basis function matrix, with each row corresponding to a parameter value. The elements consist of cubic Hermite basis functions and the scaling term h of the derivative. C is the control parameter matrix, containing the interval [missing information]. The coordinates of the two endpoints and their corresponding derivative values.

[0146] pass When calculating the coordinates of the inner and outer curve nodes, it is important to note the parameters in the two formulas. The values ​​are consistent, but the endpoint coordinates and derivative values ​​in the control parameter matrix C need to be determined according to the actual endpoint coordinates of the segmented intervals of the inner and outer curves.

[0147] For each segment interval All corresponding node pairs (outer node) With inner nodes Connect each pair of lines to create a straight line segment, and define the direction of the connecting line as the normal direction at that location. (Text height...) Determined by the average of the Euclidean distances between the first and last node pairs:

[0148]

[0149] Divide the text height evenly into ,generate Each node corresponds to a normalized parameter. ), =0 represents a point on the outer curve of the straight line segment. =1 represents a point on the inner curve of the straight line segment. For each outer node... Along its inner circle nodes Interpolation of the connection direction generates sampling points :

[0150] .

[0151] in When j=0, =0, , is the node of the outer curve; when j= hour, =1, , which is the inner curve node.

[0152] Then, bilinear interpolation is used to extract the original feature map. Extract pixel values ​​from each sampling point. coordinates First, find the four nearest integer coordinates. , , , ,(in , Select neighboring pixels. Then define the fractional offset. , The weights for bilinear interpolation are:

[0153]

[0154]

[0155] By combining the neighboring pixel values ​​and weights, the feature value is obtained:

[0156] =

[0157] + + +

[0158] Ultimately, the curved text is mapped to a fixed-size regular rectangular grid, with the length of the rectangles being... The total number of nodes in the outer curve and the pixel width Sure:

[0159]

[0160] width of the rectangle The total number of nodes between the inner and outer curves and the pixel height Sure:

[0161]

[0162] Each sampling point The final rectangular grid coordinates are:

[0163]

[0164] in The node index corresponding to the curve direction; The node index corresponding to the normal direction.

[0165] Extracted feature values Fill into the corresponding positions of the regular grid. This decouples the content (feature values) of curved text from its geometric structure (grid coordinates). The resulting regular feature map can be directly input into the subsequent recognition model without handling complex geometric deformations.

[0166] In complex industrial scenarios, existing methods fail to accurately detect text that has been bent or rotated at large angles due to inaccurate geometric modeling and insufficient rotation robustness. This invention proposes a bent and rotated text detection and recognition network based on Hermite curves and a multi-core fusion mechanism. This network can accurately and quickly perform geometric modeling of text in complex industrial scenarios. Furthermore, the text features obtained after feature alignment are regular rectangular shapes, which can be directly input into subsequent recognition models for text recognition. Therefore, this invention has significant application value for text detection in complex industrial scenarios.

[0167] This invention also provides a computer-readable storage medium storing a computer program. When executed by a computer, the computer program causes the computer to perform the steps of any of the curved and rotated text detection and recognition methods described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0168] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described curved and rotated text detection and recognition method.

[0169] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0170] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0171] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0172] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0173] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0174] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0175] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0176] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0177] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0178] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

[0179] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting and recognizing curved and rotated text, characterized in that, The method comprises the following steps executed in sequence: S1, by multi-view fusion detection mechanism, the input image containing curved or rotating text is analyzed and processed, and the optimal detection view image of the text region is generated and screened out to eliminate the detection interference caused by large angle rotation; S2, based on the optimal detection view image, a segmented cubic Hermite interpolation curve is used to parameterize the geometric modeling of the curved boundary of the text, and a smooth and continuous text contour curve is constructed; S3, using the feature alignment unit based on Hermite curve, the image features are adaptively sampled and mapped according to the text contour curve, and the curved text region is converted into a regular rectangular feature map to eliminate geometric deformation; S4, the regular rectangular feature map is input into the text recognition model, and the final text recognition result is output.

2. The method of claim 1, wherein, The multi-view fusion detection mechanism in step S1 specifically comprises the following steps: S11, processing the input image by using the instance segmentation model to generate a mask of the target object region, and extracting a minimum circumscribed rectangle region I containing the target object based on the mask crop ; S12. Based on the rotational symmetry of the target object, divide the rectangular region I... crop Dividing along its horizontal and vertical midlines, four sub-regions {R} are obtained, each covering a different aspect of the target object. top ,R bottom ,R left ,R right }; S13, a rotation transformation is applied to each of the sub-regions to make them consistent with the preset direction, so as to eliminate the text direction diversity caused by the rotation of the target object itself, and the transformed sub-regions are spliced two by two to form a plurality of view normalized spliced images; S14, the plurality of spliced images are input into the text detection model in parallel to obtain the detection result of the text in each spliced image; S15, the detection results of each spliced image are inversely transformed back to the original coordinate system, and a preset comprehensive score function S final The confidence and integrity of the text detection results under each sub-region view angle are evaluated, and finally the sub-region with the highest comprehensive score is selected as the optimal detection view angle. 3.The curved and rotated text detection and recognition method of claim 2, wherein, The implementation of step S11 specifically comprises the following steps: S11a, model application: using a SAM instance segmentation model based on deep learning, a complex industrial scene image containing curved and rotating text is processed; S11b, mask generation: using the adaptive prompting mechanism of the SAM model, an accurate binary mask M of the target object region is automatically generated; S11c, region extraction: based on the binary mask M, calculate its minimum circumscribed rectangle, and crop the rectangular region from the original input image to obtain the to-be-processed image region I containing only the target object crop to provide input for subsequent symmetric sub-region division. 4.The curved and rotated text detection and recognition method of claim 2, wherein, The comprehensive scoring function in step S15 is used to evaluate the complete coverage degree of a single sub-region to the text, and the function definition is based on the detection confidence and spatial integrity, and the expression is: ； wherein c is the average confidence of the detection frame, and the calculation formula is , c long is the confidence of the outer long text box, c short1 and c short2 are the confidences of the two short text boxes; d is the Euclidean distance from the center point of the outer long text box to the center of the target object; r is the radius of the target object; and α, β, λ are preset weight parameters.

5. The method of claim 4, wherein, When the polar distance between the center of the detection frame and the boundary of the sub-region is less than a threshold value Δθ thr , the text is determined to be a split text across adjacent sub-regions. At this point, the comprehensive score function S final A third term of text integrity weight is introduced, and the function expression is extended as follows: ； wherein: L local is the arc length of the outer contour of the bounding box in the current sub-region calculated by the piecewise cubic Hermite interpolation curve; L global is the estimated total arc length of the global outer contour of the bounding box, and the value is the sum of the arc lengths L local of the segmented text in each adjacent sub-region; and γ is a preset integrity weight coefficient. 6.The curved and rotated text detection and recognition method of claim 1, wherein, In step S2, the segmented cubic Hermite interpolation curve is used to parameterize the geometric modeling of the text boundary, and the specific implementation comprises the following steps: S21, Curve parameterization definition: Represent the geometry of the text boundary as coordinate functions in terms of an independent parameter t, for a sequence of nodes partitioned intervals where i = 0, 1,..., n - 1, define a local parameter s to control the single-segment curve interpolation position: ； wherein is the interval length; S22, basis function construction: the function values and first derivative values at the nodes are combined by using the following defined cubic Hermite basis function to ensure the first derivative continuity of the curve at the nodes: ； ； ； ； wherein, is a start function value weight, is a start derivative value weight, is an end function value weight, is an end derivative value weight; S23. Curve segment equation generation: Based on the base functions, the parametric equations of the curve coordinate components x(t) and y(t) are determined by the endpoint coordinates and their derivative values within the interval ​ ； where m xi , m yi are the first order parameter derivatives at the node P i = (x i , y i ); S24, overall curve synthesis: synthesizing the interval curve P(t) = (x(t), y(t)) T in terms of the parameter t∈ sequentially connecting to form a complete smooth text boundary curve.

7. The method of claim 6, wherein, The step S23 of calculating the node parameter derivative m xi The calculation method of m yi is as follows: S231, internal node derivative calculation: for the internal node P which is not an endpoint i (i = 1, 2, …, n-1), to ensure that the adjacent curve segments are consistent in the tangent direction at the node, the derivative is calculated using the central difference method, and the formula is: ； S232, endpoint node derivative calculation: for the starting node P0 and the ending node Pn, the derivative is calculated by using the one-sided difference method, and the formula is: ； ； S233, derivative continuity guarantee: the internal nodes are given a unified derivative value by using the central difference method to ensure the first derivative continuity of the curve at the internal nodes, thereby ensuring the smoothness of the overall contour curve. 8.The curved and rotated text detection and recognition method of claim 1, wherein, The feature alignment unit based on Hermite curve in step S3 performs lossless mapping of the curved text region to the regular feature space according to the following steps: S31, outer circle curve length calculation and sampling point parameter generation: for the outer circle curve of the text contour, numerical integration is performed on each segmented interval [t i ,t i+1 ] of the outer circle curve to calculate the actual arc length L outer,i : ； and evenly divide the arc length into parts, generate a normalized parameter sequence s k = k / L outer,i , where k = 0, 1, …, L outer,i , s0= 0 corresponds to the start of the segment, and s Louter,i = 1 corresponds to the end of the segment; S32, Inner and outer ring curve node coordinate synchronization calculation: using piecewise cubic Hermite interpolation matrix The inner and outer ring curve node coordinates are obtained, and the matrix expression of the piecewise cubic Hermite interpolation curve is obtained As follows: ； The inner curve is forced to use the same number of segments and parameter sequence s as the outer curve. k The inner and outer curves are calculated using the matrix expression of the piecewise cubic Hermite interpolation curves with respect to parameter s. k The coordinates of the interpolation point at: ； ； Where P is the path point coordinate matrix, and each row corresponds to a parameter. The interpolation point coordinate vector at ( , B is a basis function matrix, with each row corresponding to a parameter value. The elements consist of cubic Hermite basis functions and the scaling term h of the derivative, meaning each row of elements consists of basis functions h. 00 (s k ), h 01 (s k ), h h 10 (s k ), h h 11 (s k The matrix is ​​composed of ( ), where C is the control parameter matrix and its row vectors are the corresponding intervals. The coordinates of the endpoints and their derivative (x) i ,y i ,m xi ,m yi ) T ; S33, text height calculation and normal direction sampling parameter generation: for each pair of inner and outer circle nodes (P outer (s k ),P inner (s k )), calculate the text height H i there as the average of the Euclidean distance between the head and tail nodes. ； and the height direction is evenly divided into H i The height direction normalization parameter is generated ), v0=0 corresponds to the point on the outer circle curve, and v Hi =1 corresponds to the point on the inner circle curve. S34, orthogonal sampling grid construction: for each parameter pair (s k ,v j ), interpolate along the direction of the inner and outer ring node connecting line to calculate the coordinates of the sampling point Q. ； S35, bilinear interpolation feature extraction and regular grid filling: for the coordinates (x k,j ,y k,j ) of each sampling point Q(k,j), the feature value is extracted from the original feature map F through bilinear interpolation method, and is filled into the corresponding position of the regular rectangular grid , where Δw, Δh are preset pixel intervals; finally, a regular feature map with size l×w is generated, where l=(L outer +1) Δ w , w=(H i +1) Δh.

9. The method of claim 8, wherein, The bilinear interpolation method in step S35 is executed according to the following specific steps to realize high-fidelity feature mapping from the original feature map to the regular grid: S351. Nearest-neighbor integer coordinate lookup: For the floating-point coordinates (xk,j, yk,j) of any sample point Q(k,j), first find the four nearest-integer-coordinate pixel points , , , where , , denotes the floor operation: S352, decimal offset calculation: the decimal offset (Δx, Δy) between the sampling point coordinates and the reference integer coordinates (a, b) is calculated, and the calculation formula is: , ； S353、Bilinear weight calculation: according to the decimal offset (Δx, Δy), the bilinear interpolation weight corresponding to four neighborhood pixel points is calculated, and the calculation formula is as follows: ； ； ； ； S354, feature value fusion: according to the calculated weight, the feature values of the four neighboring pixel points are weighted and summed to obtain the final interpolation feature value F of the sampling point Q(k,j) align(k,j) : = + + + ； wherein, represents the feature value at the corresponding coordinate on the original feature map.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the computer. When the computer executes the steps in any one of the curved rotating text detection and recognition methods of claims 1-9.

Images