Deformed text detection method and device, and storage medium

Abstract

The application discloses a morphing text detection method and device and a storage medium. The method comprises the following steps: extracting image features of a to-be-processed image, the to-be-processed image being an image collected from a specified text carrier, and the specified text carrier being easy to deform under stress; inputting the image features into a control point detection network to obtain an edge control point sequence; inputting the edge control point sequence into a differentiable B-spline layer to perform matrix multiplication calculation on a base function matrix and the edge control point sequence, thereby obtaining a B-spline curve point sequence, and the base function matrix being a two-dimensional matrix composed of B-spline base function values; and generating a text detection frame based on the B-spline curve point sequence. The operation of the differentiable B-spline layer is differentiable in a deep learning framework, and the gradient can be back propagated to the parameters of the control point detection network through the differentiable B-spline layer, thereby realizing integrated end-to-end optimization of text detection and curve fitting and significantly improving the performance of the model in a complex morphing text detection task.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, device and storage medium for detecting deformed text. Background Technology

[0002] Text serves as an important carrier of information, such as banners and streamers hanging outdoors, which carry important semantic information. Image text recognition technology is a technique for extracting text content from images.

[0003] Before performing text recognition, text region detection is required. Ordinary text detectors can only detect text content arranged in a regular pattern and have difficulty detecting text with complex deformations, such as banners and streamers, which have complex bending deformations due to physical factors such as gravity, wind pulling, and asymmetrical suspension. This makes it difficult for traditional detection boxes to accurately cover the text region. While Bézier curves or polygons can be used to detect curved text, the algorithm is still susceptible to noise interference such as wrinkles, shadows, and highlights, and is prone to severe jitter in the predicted curve path, resulting in text distortion after correction and thus recognition errors. Although neural network models can jointly optimize detection and recognition, their generalization ability is weak, and they suffer from low edge regression accuracy and jagged effects.

[0004] Therefore, how to improve the accuracy of text detection in complex scenarios is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides at least one method, device, and storage medium for detecting deformed text.

[0006] The first aspect of this application provides a method for detecting deformed text. The method includes: extracting image features from an image to be processed, wherein the image to be processed is an image acquired from a specified text carrier, and the specified text carrier is easily deformed under force; inputting the image features into a control point detection network to identify the coordinates of edge control points to obtain an edge control point sequence; wherein the edge control points refer to the points on the edges of the text region in the image to be processed, and the edge control point sequence is represented as a two-dimensional matrix; inputting the edge control point sequence into a differentiable B-spline layer to perform matrix multiplication calculation on the basis function matrix and the edge control point sequence to obtain a B-spline curve point sequence; wherein the basis function matrix refers to a two-dimensional matrix obtained by combining the B-spline basis function values; and generating a text detection box based on the B-spline curve point sequence.

[0007] In one embodiment, the method further includes: calculating the model training loss based on the B-spline curve point sequence; calculating the gradient of the model training loss with respect to the B-spline curve point sequence, and backpropagating the gradient through a differentiable B-spline layer to update the control point detection network.

[0008] In one embodiment, the edge control point sequence includes an upper edge point sequence and a lower edge point sequence. The upper and lower edge point sequences are input into a differentiable B-spline layer, respectively, resulting in a B-spline curve point sequence including an upper edge curve point sequence and a lower edge curve point sequence. Based on the B-spline curve point sequence, the model training loss is calculated, including: calculating the curvature of the upper edge curve at each sampling point based on the upper edge curve point sequence to obtain the upper edge curvature; calculating the curvature of the lower edge curve at each sampling point based on the lower edge curve point sequence to obtain the lower edge curvature; calculating the curvature smoothing loss based on the upper and lower edge curvatures; and / or calculating the spacing change rate between the upper and lower edge curves based on the spacing between the points in the upper and lower edge curve point sequences; calculating the edge parallelism loss based on the spacing change rate; and calculating the model training loss based on the curvature smoothing loss and / or the edge parallelism loss.

[0009] In one embodiment, the control point detection network includes a convolutional layer, an activation function, an upsampling layer, and a location extraction layer. Inputting image features into the control point detection network to identify the coordinates of edge control points and obtain an edge control point sequence includes: inputting image features into the convolutional layer to obtain convolutional features output by the convolutional layer, where the number of channels in the convolutional features is equal to the number of edge control points to be identified; inputting the convolutional features into the activation function to obtain activation features output by the activation function, where the values ​​in the activation features represent normalized coordinates; upsampling the activation features based on the upsampling layer to obtain upsampled features; and inputting the upsampled features into the location extraction layer to determine the maximum response coordinates of each channel, thereby obtaining the edge control point sequence output by the location extraction layer.

[0010] In one embodiment, the method further includes: obtaining a preset clamped node vector; recursively calculating all B-spline basis function values ​​based on the clamped node vector; and organizing all B-spline basis function values ​​into a two-dimensional matrix to obtain the basis function matrix.

[0011] In one embodiment, the edge control point sequence includes an upper edge point sequence and a lower edge point sequence. The upper edge point sequence and the lower edge point sequence are respectively input into a differentiable B-spline layer, and the resulting B-spline curve point sequence includes an upper edge curve point sequence and a lower edge curve point sequence. Generating a text detection box based on the B-spline curve point sequence includes: calculating the short sides on both sides of the upper edge curve and the lower edge curve based on the upper edge curve point sequence and the lower edge curve point sequence; and generating a text detection box by combining the upper edge curve, the lower edge curve, and the short sides.

[0012] In one embodiment, the starting point in the B-spline curve point sequence is equal to the starting edge control point in the edge control point sequence, and the ending point in the B-spline curve point sequence is equal to the ending edge control point in the edge control point sequence. Based on the upper edge curve point sequence and the lower edge curve point sequence, the short sides on both sides of the upper edge curve and the lower edge curve are calculated, including: calculating the tangent vectors corresponding to the starting points of the upper edge curve point sequence and the lower edge curve point sequence to obtain a first tangent vector; calculating the tangent vectors corresponding to the ending points of the upper edge curve point sequence and the lower edge curve point sequence to obtain a second tangent vector; and generating the short sides on both sides of the upper edge curve and the lower edge curve based on the first tangent vector and the second tangent vector, respectively.

[0013] In one embodiment, the method further includes: correcting the text content in the image to be processed based on the text detection box to obtain a corrected image to be processed; and performing text recognition on the corrected image to obtain a text recognition result.

[0014] A second aspect of this application provides a deformable text detection device, comprising: a feature extraction module for extracting image features of an image to be processed, wherein the image to be processed is an image acquired from a specified text carrier, the specified text carrier being easily deformable under force; a control point detection module for inputting the image features into a control point detection network to identify the coordinates of edge control points, thereby obtaining an edge control point sequence; wherein the edge control points refer to the points at the edges of the text region in the image to be processed, and the edge control point sequence is represented as a two-dimensional matrix; a curve calculation module for inputting the edge control point sequence into a differentiable B-spline layer to perform matrix multiplication calculation on the basis function matrix and the edge control point sequence, thereby obtaining a B-spline curve point sequence; wherein the basis function matrix refers to a two-dimensional matrix obtained by combining the B-spline basis function values; and a detection box generation module for generating text detection boxes based on the B-spline curve point sequence.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-described deformed text detection method.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described deformed text detection method.

[0017] The above scheme extracts image features from the image to be processed; inputs these features into a control point detection network to identify the coordinates of edge control points, obtaining an edge control point sequence; inputs this edge control point sequence into a differentiable B-spline layer to perform matrix multiplication on the basis function matrix and the edge control point sequence, obtaining a B-spline curve point sequence; where the basis function matrix is ​​a two-dimensional matrix obtained by combining the B-spline basis function values; and generates text detection boxes based on the B-spline curve point sequence. By using matrix multiplication in the differentiable B-spline layer to combine the control point coordinates with the basis function matrix to generate a point sequence on the curve, the computation process is differentiable within the deep learning framework. The gradient can be backpropagated to the parameters of the control point detection network through the differentiable B-spline layer, achieving integrated end-to-end optimization of text detection and curve fitting, significantly improving the model's performance on complex deformed text detection tasks.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a flowchart illustrating a deformed text detection method in an exemplary embodiment of this application; Figure 2 This is a schematic diagram illustrating text detection of a banner image according to an embodiment of this application; Figure 3 This is a block diagram illustrating a deformed text detection device according to an exemplary embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0021] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0022] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0023] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] The deformed text detection method provided in the embodiments of this application will be described below.

[0025] The deformed text detection method provided in this application can be executed by a terminal, a server, or by the interaction between the terminal and the server. That is, some steps of the method are executed by the terminal and other steps are executed by the server. This application does not limit the type of execution subject of the deformed text detection method.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating a deformed text detection method in an exemplary embodiment of this application, as shown below. Figure 1 As shown, the deformed text detection method includes at least steps S110 to S140, which are described in detail below: Step S110: Extract image features of the image to be processed, wherein the image to be processed is an image acquired from a specified text carrier, and the specified text carrier is easily deformed under force.

[0027] The specified text carrier includes, but is not limited to, banners, streamers, flags, etc. These types of specified text carriers are easily deformed under force, which causes the text on the specified text carrier to be easily deformed.

[0028] For example, banners and streamers may exhibit wavy, S-shaped, or partially folded forms due to physical factors such as gravity, wind pulling, or asymmetrical hanging, which in turn causes the text on them to appear complexly curved and deformed.

[0029] The image to be processed can be an image taken by a camera of a designated text carrier such as banners, streamers, and flags hanging on the road. This application does not limit the type of image to be processed.

[0030] Optionally, the image to be processed is a preprocessed image, including but not limited to image enhancement, and / or image cropping, and / or resizing, and / or pixel normalization.

[0031] Taking banner text recognition as an example, the process involves capturing images from a high-definition video stream to obtain original image frames. A banner detection algorithm is then used to detect the banner area in the original image frames. The image content of the banner area is extracted from the original image frames to obtain a small banner image. The small banner image is then scaled to a fixed size, and the pixels are normalized to the range of [0,1] to obtain the image to be processed.

[0032] After obtaining the image to be processed, extract the image features of the image to be processed.

[0033] Image features include, but are not limited to, one or more of the following: color features, texture features, shape features, spatial relationship features, and depth features.

[0034] For example, Histogram of Oriented Gradient (HOG) and Scale-invariant feature transform (SIFT) can be used to extract the shape and texture of an image; Convolutional Neural Networks (CNNs) (such as Feature Pyramid Network (FPN) and Residual Network (ResNet)) or Vision Transformers (ViTs) can be used as backbone networks to encode the input image to be processed and obtain image features.

[0035] For example, image features of the image to be processed are extracted through a backbone network and a feature fusion module. The backbone network uses FPN to extract multi-scale features to address text scale variations, and the feature fusion module uses Path Aggregation Network (PANet) to enhance semantic fusion, that is, to enhance the representation ability of the backbone network by fusing bottom-up and top-down paths.

[0036] Step S120: Input the image features into the control point detection network to identify the coordinates of edge control points and obtain the edge control point sequence; where edge control points refer to the points on the edges of the text region in the image to be processed, and the edge control point sequence is represented as a two-dimensional matrix.

[0037] Control point detection networks are used to identify the coordinates of points on the edge contours of text line regions in an image to be processed.

[0038] For example, the output of a control point detection network is the coordinates of 2K edge control points, including K edge control points on the upper edge and K edge control points on the lower edge.

[0039] Of course, in addition to outputting edge control points for multiple edges, it is also possible to output edge control points for a single edge. The specific settings can be flexibly configured according to the actual application scenario, and this application does not impose any limitations on this.

[0040] Based on the coordinates of the identified edge control points, an edge control point sequence is obtained. This edge control point sequence is represented as a two-dimensional matrix to facilitate matrix calculations in subsequent differentiable B-spline layers.

[0041] Step S130: Input the edge control point sequence into the differentiable B-spline layer to perform matrix multiplication calculation on the basis function matrix and the edge control point sequence to obtain the B-spline curve point sequence; where the basis function matrix refers to the two-dimensional matrix obtained by combining the B-spline basis function values.

[0042] It can calculate the basis function values ​​corresponding to openB spline curves, dlampedB spline curves, or closedB spline curves.

[0043] Among them, OpenB spline curves do not necessarily pass through the first and last control points, and the node vectors are usually uniformly distributed; ClampedB spline curves make the curve pass through the first and last control points precisely by repeating nodes, and the node vector characteristics are: the nodes at both ends are repeated p+1 times (p is the number of times); ClosedB spline curves form a closed loop by connecting the first and last ends, and use uniform or quasi-uniform node vectors.

[0044] The node vector is a non-decreasing sequence of real numbers that determines how parameters are mapped to curve segments.

[0045] Basis functions are a set of piecewise polynomial functions defined based on node vectors. Control points are weighted and summed using basis functions to form curves.

[0046] The computation process in traditional text detection post-processing (such as OpenCV's approxPolyDP) is usually non-differentiable and cannot participate in gradient backpropagation. Therefore, during the training phase, the gradient cannot be propagated from the final curve error to the control point coordinates, which leads to a disconnect between text detection and curve fitting, resulting in low text detection accuracy.

[0047] To address the aforementioned issues, this application combines the B-spline basis function values ​​into a two-dimensional matrix, performs matrix multiplication on the basis function matrix and the edge control point sequence to obtain the B-spline curve point sequence, and constructs a differentiable B-spline layer. Thus, during the model training phase, the gradient can propagate to the control point detection network through the differentiable B-spline layer, enabling end-to-end learning of control point regression, thereby significantly improving the model's performance on complex deformed text detection tasks.

[0048] Step S140: Generate a text detection box based on the B-spline curve point sequence.

[0049] Based on the B-spline curve point sequence, the text content region in the image to be processed is determined, and the corresponding text detection box is generated.

[0050] For example, a B-spline curve point sequence includes an upper edge curve point sequence and a lower edge curve point sequence. This can be achieved by obtaining the curves corresponding to the upper edge curve point sequence and the lower edge curve point sequence, respectively, to obtain the upper edge curve and the lower edge curve. Then, based on the tangent direction of the curve endpoints or by directly connecting the endpoints of the two curves, short sides are generated on both sides to obtain the text detection box.

[0051] For example, a B-spline curve point sequence includes an upper edge curve point sequence and a lower edge curve point sequence. A text detection box can be obtained by connecting all the points of the upper edge curve point sequence and the lower edge curve point sequence in sequence.

[0052] For example, if the B-spline curve point sequence only contains a single long side curve point sequence, then the curve corresponding to the curve point sequence can be obtained, or the points in the curve point sequence can be connected in order to obtain a single long border. Then, based on the predicted font height and the direction of the font relative to the single long border, another long border can be generated. Connecting the endpoints of the two long borders generates the two short sides, resulting in a text detection box.

[0053] The top and bottom edges in the above examples are merely illustrative examples, referring to the longer side of the text. In actual application scenarios, for vertically arranged text, the top and bottom edges correspond to the left and right edges.

[0054] It should be noted that the above steps can be performed during model training or model inference, and this application does not limit them in this regard.

[0055] If the above steps are executed during the model inference process, after obtaining the text detection box, text recognition can be performed based on the text detection box to obtain the specific text content contained in the image to be processed. Alternatively, the text detection box can be directly output to show the region where the text content is located in the image to be processed.

[0056] If the above steps are performed during model training, the model can be trained based on the text detection boxes and / or edge control point sequences. For details of the training process, please refer to the following examples.

[0057] This application uses differentiable B-splines for text curve fitting. Since each control point only affects a local interval of the curve, moving a control point will only affect a part of the curve segment near the B-spline curve, without changing the shape of the entire curve. This solves the problem that a single point change in the Bézier curve affects the entire edge. Moreover, the generated curve is more in line with the real physical hanging law of the specified text carrier such as banners and strips, making the generated curve more stable and accurate.

[0058] This application does not treat the curve calculation process as a non-differentiable post-processing step. Instead, it constructs a differentiable B-spline layer. This layer uses matrix multiplication to combine the control point coordinates with the basis function matrix to generate a sequence of points on the curve. This series of operations is differentiable in the deep learning framework, allowing the gradient to propagate back from the final curve quality to the parameters of the control point detection network through the differentiable B-spline layer. The control point detection network can learn how to directly predict the control points that can generate the optimal curve through the differentiable B-spline layer. This achieves integrated end-to-end optimization of text detection and curve fitting, significantly improving the model's performance on complex deformed text detection tasks, especially enhancing the anti-interference ability of curved text detection on hanging banners. It also ensures fast detection speed and real-time text detection.

[0059] The following describes some embodiments of this application in detail.

[0060] In some implementations, the control point detection network includes a convolutional layer, an activation function, an upsampling layer, and a location extraction layer; in step S120, image features are input into the control point detection network to identify the coordinates of edge control points to obtain an edge control point sequence, including steps S121 to S124.

[0061] Step S121: Input the image features into the convolutional layer to obtain the convolutional features output by the convolutional layer. The number of channels in the convolutional features is equal to the number of edge control points to be identified.

[0062] If the number of edge control points to be identified is 2K, then the input of the convolutional layer is the image features, and the output is the convolutional features with 2K channels. Each edge control point corresponds to one channel, and each edge control point has two values: the horizontal coordinate (x) and the vertical coordinate (y).

[0063] The kernel size of the convolutional layer can be flexibly set according to the actual situation. For example, the kernel size of the convolutional layer is 1×1. This application does not limit this.

[0064] Step S122: Input the convolutional features into the activation function to obtain the activation features output by the activation function. The values ​​in the activation features represent the normalized coordinates.

[0065] The activation function can be Sigmoid, Softmax, ReLU, etc., and this application does not limit it.

[0066] Taking the Sigmoid activation function as an example, the initial response value of each channel output by the previous convolutional layer is mapped to the interval [0, 1] through the Sigmoid function. After Sigmoid processing, each output value represents the normalized coordinate ratio relative to the current feature map size.

[0067] Step S123: Upsample the activation features based on the upsampling layer to obtain upsampled features.

[0068] Then, the smaller feature map, after being activated by the activation function, is upsampled back to the size of the original input image.

[0069] Step S124: Input the upsampled features into the location extraction layer to determine the maximum response coordinates of each channel, and obtain the edge control point sequence output by the location extraction layer.

[0070] The coordinates of the control points are finally determined based on the upsampling features.

[0071] For example, the upsampled feature corresponds to a response map with 2K channels, and each channel corresponds to a specific coordinate value (such as the coordinates of the first control point at the top edge). On the response map of each channel, the position of the pixel with the largest response value is found, and the position of the pixel with the largest response value is taken as the maximum response coordinate of that channel.

[0072] It should be noted that the above K values ​​are based on experience, such as K can be 3, 4, 5, 6, etc.

[0073] The above embodiments, through a lightweight control point detection network, regress only a small number of control points, significantly reducing the number of model parameters and computational complexity, and avoiding overfitting. Then, the sequence of edge control points is converted into a sequence of points of the curve through a differentiable B-spline layer.

[0074] Taking a total of 2K edge control points at the top and bottom edges as an example, the dimension of the edge control point sequence for the top and bottom edges is batch size × number of edge control points × 2. Differentiable B-spline layers receive independent edge control point sequences for each edge. As input, the number of edge control points (K=n+1) is 3 to 6, preferably 4 or 5. The upper and lower edges are processed using differentiable B-spline layers with the same parameters.

[0075] The type of B-spline curve used in the differentiable B-spline layer can be an Open B-spline curve, a Clamped B-spline curve, or a Closed B-spline curve. The specific type can be flexibly selected according to the actual application scenario, and this application does not limit it.

[0076] For example, to ensure the curve is forced to pass through the first and last control points, a Clamped B-spline curve is generated. Specifically, a preset clamped node vector is obtained; all B-spline basis function values ​​are recursively calculated based on the clamped node vector; and all B-spline basis function values ​​are organized into a two-dimensional matrix to obtain the basis function matrix.

[0077] To generate B-spline curves, a clamped node vector is preset. The node vector is a non-decreasing sequence, and its length is the number of edge control points plus the B-spline order p plus 1. The value of p can be flexibly selected according to the actual application scenario, and this application does not limit it.

[0078] For example, for a cubic B-spline (order p=3), the first four and last four elements of the clamped node vector are filled with zeros and ones respectively, and the middle segment is uniformly distributed between 0 and 1. The specific construction is as follows: Formula 1: (Formula 1) Where n is the number of edge control points minus 1 (i.e., the index of the last edge control point), ensuring that the clamped property forces the curve to pass through the first and last control points.

[0079] Furthermore, a parameter sequence is generated, containing multiple parameter values ​​sampled from a valid interval. For example, the parameter values ​​in the parameter sequence are uniformly sampled within the interval [0, 1], with a sampling interval Δt = 0.05 and a sampling number of 21 points (corresponding to t = 0, 0.05, 0.1, ..., 1.0), used for parameterizing the B-spline curve. The sampling interval Δt = 0.05 has been experimentally verified to achieve the best balance between accuracy and computational efficiency. Of course, other sampling intervals or sampling methods can be selected according to the actual application scenario. For example, in addition to uniform parameterization, chord length parameterization can also be used; this application does not impose any limitations on this.

[0080] The clamped node vector ensures that, at parameter t=0, the curve point P(0) is exactly equal to the first edge control point. When the parameter t=1, the curve point P(1) is exactly equal to the last edge control point. The midpoint forms a smooth curve, reflecting the true shape of the text area.

[0081] Then, the B-spline basis functions are computed. For each parameter value and the index of each edge control point, a recursive algorithm (such as the de Boor recursive algorithm, the de Casteljau recursive algorithm, etc.) is used to compute the corresponding basis function value. Specifically, the B-spline basis functions... Calculate recursively using the following formulas 2 and 3: (Formula 2) (Formula 3) When the denominator is zero, the corresponding term is set to zero. This calculation process is completely differentiable and supports gradient backpropagation.

[0082] Through the above steps, the values ​​of all corresponding B-spline basis functions are calculated for each sampling point.

[0083] It should be noted that the above method for calculating the B-spline basis function value is only an example. Other types of B-spline curves can also be used to calculate the B-spline basis function value, and this application does not limit this.

[0084] The calculated B-spline basis function values ​​are organized into a two-dimensional matrix to obtain the basis function matrix. The dimension of the basis function matrix is ​​the number of sampling points × the number of edge control points. For example, for a clamped cubic B-spline, the basis function matrix has 21 rows (corresponding to 21 sampling points) and n+1 columns (corresponding to n+1 edge control points). The elements in the matrix... This represents the basis function value corresponding to the i-th edge control point at the k-th sampling point.

[0085] The basis function matrix reflects the influence weight of each edge control point on different positions on the curve.

[0086] basis function matrix With edge control point sequence Perform matrix multiplication to obtain the B-spline curve point sequence. Its dimension is batch size × number of sampling points × 2, meaning that for each sample, a sequence of (x, y) coordinates of 21 sampling points on the curve is obtained. , It is the discrete representation of the fitted B-spline curve.

[0087] In the aforementioned differentiable B-spline layer, the recursive algorithm's operations are themselves differentiable, as is the calculation of the basis function matrix. Therefore, the entire process of generating the basis function matrix is ​​differentiable, resulting in the final B-spline curve point sequence. It is a basis function matrix With edge control point sequence The result of matrix multiplication is also a differentiable operation. Therefore, from the B-spline curve point sequence To input edge control point sequence The gradient path is completely unobstructed, and the loss function can be applied during the model training phase. The quality is calculated, and the gradient is backpropagated to the control point detection network through the differentiable B-spline layer, thereby guiding the control point detection network to learn how to predict the edge control points that can generate high-quality curves through the differentiable B-spline layer, thus achieving end-to-end training.

[0088] Specifically, during the model training phase, the model training loss is calculated based on the B-spline curve point sequence; the gradient of the model training loss with respect to the B-spline curve point sequence is calculated, and the gradient is backpropagated through a differentiable B-spline layer to update the control point detection network.

[0089] The quality of the B-spline curve point sequence is evaluated to calculate the model training loss. The worse the quality of the B-spline curve point sequence, the greater the model training loss, and the better the quality of the B-spline curve point sequence, the smaller the model training loss.

[0090] For example, the curvature smoothness and / or parallelism of two opposite edges can be calculated from the B-spline curve point sequence. The quality of the B-spline curve point sequence can be evaluated based on the curvature smoothness and / or the parallelism of two opposite edges to obtain the model training loss.

[0091] Taking the upper and lower edges as an example: the edge control point sequence includes an upper edge point sequence and a lower edge point sequence. These sequences are input into a differentiable B-spline layer, resulting in a B-spline curve point sequence including an upper edge curve point sequence and a lower edge curve point sequence. Based on the B-spline curve point sequence, the model training loss is calculated, including: calculating the curvature of the upper edge curve at each sampling point based on the upper edge curve point sequence; calculating the curvature of the lower edge curve at each sampling point based on the lower edge curve point sequence; calculating the curvature smoothing loss based on the upper and lower edge curvatures; and / or calculating the rate of change of the distance between the upper and lower edge curves based on the distance between the points in the upper and lower edge curve point sequences; calculating the edge parallelism loss based on the rate of change of the distance; and calculating the model training loss based on the curvature smoothing loss and / or the edge parallelism loss.

[0092] Among them, curvature smoothing loss is used to constrain the generated B-spline curve to have a lower curvature change, so as to suppress unnatural high-frequency jitter, make the curve smoother, and make the curve conform to the physical suspension law.

[0093] Edge parallelism loss is used to constrain the two opposite edges of a text line to remain roughly parallel geometrically, making the detected text regions more realistic.

[0094] Specifically, let the point sequence of the B-spline curve be... ,in If there are T sampled points, then the curvature of any curve at each point t is calculated using the following formula 4. : (Formula 4) in, , The first and second numerical derivatives of the x-axis with respect to t (can be achieved using torch.gradient). , Let be the first and second numerical derivatives of the ordinate with respect to t.

[0095] Then define curvature smoothing loss Formula 5 is as follows: (Formula 5) in, Indicates the curvature of the upper edge. Indicates the curvature of the lower edge.

[0096] In addition, the spacing between the points in the upper edge curve point sequence and the lower edge curve point sequence is calculated. The specific calculation formula is shown in Formula 6 below: (Formula 6) in, For k points in the upper edge curve point sequence, For k points in the lower edge curve point sequence, This represents the distance between k points in the upper edge curve point sequence and the lower edge curve point sequence.

[0097] Then, the rate of change of the spacing at the k-th point is calculated based on the following formula 7. : (Formula 7) Then define edge parallelism loss. Formula 8 is as follows: (Formula 8) By incorporating curvature smoothing loss, high-frequency noise (such as "Z"-shaped jitter) can be avoided during edge control point detection.

[0098] By using edge parallel loss, the spacing between adjacent edge points of two edge curves is forced to change gradually, which conforms to the geometric characteristics of real text regions, ensuring smooth upper and lower edges and maintaining the integrity of the text region in wrinkled areas.

[0099] Optionally, in addition to calculating the model training loss based on the B-spline curve point sequence, the model training loss can also be calculated based on the edge control point sequence output by the control point detection network. For example, the expected coordinates of the control points marked in the sample image to be processed can be obtained, and the L1 loss can be calculated based on the difference between the actual coordinates and the expected coordinates of the control points in the edge control point sequence to obtain the control point coordinate loss. The model training loss is calculated by combining the curvature smoothing loss, the edge parallelism loss, and the control point coordinate loss. For details, please refer to Formula 9 below: (Formula 9) in, , These are dynamic weighting coefficients, which can be flexibly set according to the actual application scenario, such as... The value is 0.6. The value is 0.3.

[0100] Of course, any one of the curvature smoothing loss, the edge parallelism loss, and the control point coordinate loss can be used as the model training loss, or any two of the curvature smoothing loss, the edge parallelism loss, and the control point coordinate loss can be weighted and summed to obtain the model training loss. This application does not limit this.

[0101] The model training loss calculated above can be propagated through a differentiable B-spline layer to train the control point detection network, improve the accuracy of the edge control point sequence output by the control point detection network, and achieve accurate detection of complex deformed text.

[0102] The differentiable B-spline layer calculates the B-spline curve point sequence based on the edge control point sequence output by the control point detection network, and generates a text detection box based on the B-spline curve point sequence.

[0103] Similarly, taking the upper and lower edges as examples, the edge control point sequence includes the upper edge point sequence and the lower edge point sequence. The upper edge point sequence and the lower edge point sequence are respectively input into the differentiable B-spline layer, and the resulting B-spline curve point sequence includes the upper edge curve point sequence and the lower edge curve point sequence. In step S140, a text detection box is generated based on the B-spline curve point sequence, including steps S141 to S142.

[0104] Step S141: Based on the upper edge curve point sequence and the lower edge curve point sequence, calculate the short sides on both sides of the upper edge curve and the lower edge curve.

[0105] For example, using a clamped B-spline, the starting point in the B-spline curve point sequence is equal to the starting edge control point in the edge control point sequence, and the ending point in the B-spline curve point sequence is equal to the ending edge control point in the edge control point sequence. Based on the upper edge curve point sequence and the lower edge curve point sequence, the short sides on both sides of the upper edge curve and the lower edge curve are calculated, including: calculating the tangent vectors corresponding to the starting points of the upper edge curve point sequence and the lower edge curve point sequence to obtain the first tangent vector; calculating the tangent vectors corresponding to the ending points of the upper edge curve point sequence and the lower edge curve point sequence to obtain the second tangent vector; and generating the short sides on both sides of the upper edge curve and the lower edge curve based on the first tangent vector and the second tangent vector, respectively.

[0106] The clamped B-spline algorithm guarantees the starting point Left vertex, end point The right vertex; A detailed explanation of the calculation process for the left shorter side is provided below: The first tangent vector is obtained by calculating the tangent vector at the left endpoint using the following formula 10: (Formula 10) Then, the vertical boundary is generated to obtain the short side, as detailed in Formula 11 below: (Formula 11) Where x refers to the horizontal coordinate and y refers to the vertical coordinate.

[0107] In addition, to determine the length of the shorter side, please refer to Formula 12 below: (Formula 12) in, This is the preset maximum text height.

[0108] Similarly, the right-side shorter side is calculated in the same way.

[0109] Step S142: Combine the upper edge curve, lower edge curve, and short side to generate a text detection box.

[0110] The text detection box is obtained by combining the closed shape formed by the upper edge curve, the lower edge curve, and the two short sides.

[0111] In addition, for vertically arranged text, the above-mentioned top and bottom edges are reinterpreted as left and right edges, and the short edge generation mechanism is migrated synchronously without modifying the network structure.

[0112] For example, in the case of banner text recognition, the aspect ratio of the banner is calculated. If the ratio is less than 1.5, the top and bottom edges are reinterpreted as left and right edges, and the coordinate axes are swapped while keeping the short side generation mechanism unchanged, thus achieving vertical text detection.

[0113] In some implementations, the method further includes: correcting the text content in the image to be processed based on the text detection box to obtain a corrected image to be processed; and performing text recognition on the corrected image to obtain a text recognition result.

[0114] Due to the curvature of the object's surface or the shooting angle, the text within the text detection box may be tilted, trapezoidal, or curved. The correction process will correct the distorted text area into a regular rectangular image based on the geometry of the text detection box through algorithms such as perspective transformation or thin plate spline transformation.

[0115] Then, the corrected regular rectangular image is fed into a pre-trained text recognition model for text recognition, and the corresponding text sequence is output to obtain the text recognition result.

[0116] For a detailed explanation using banner text detection as an example, please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram illustrating text detection of a banner image, as shown in one embodiment of this application. Figure 2As shown, the input is a banner image, which is preprocessed. The preprocessed image is then input into a feature extraction network to extract image features. These features are then input into a control point detection network to identify the coordinates of edge control points, resulting in an edge control point sequence. This sequence is then input into a differentiable B-spline layer for curve fitting, yielding a B-spline curve point sequence. Finally, text line generation post-processing is performed based on the B-spline curve point sequence to generate text detection boxes, which are then output as the final text detection boxes.

[0117] Figure 3 This is a block diagram illustrating a deformed text detection device according to an exemplary embodiment of this application. Figure 3 As shown, the exemplary deformed text detection device 300 includes: The feature extraction module 310 is used to extract image features of the image to be processed, wherein the image to be processed is an image acquired from a specified text carrier, and the specified text carrier is easily deformed under force; The control point detection module 320 is used to input image features into the control point detection network to identify the coordinates of edge control points and obtain an edge control point sequence; wherein, the edge control point refers to the point on the edge of the text region in the image to be processed, and the edge control point sequence is represented as a two-dimensional matrix; The curve calculation module 330 is used to input the edge control point sequence into the differentiable B-spline layer to perform matrix multiplication calculation on the basis function matrix and the edge control point sequence to obtain the B-spline curve point sequence; wherein, the basis function matrix refers to the two-dimensional matrix obtained by combining the B-spline basis function values. The detection box generation module 340 is used to generate text detection boxes based on the B-spline curve point sequence.

[0118] It should be noted that the deformed text detection device and the deformed text detection method provided in the above embodiments belong to the same concept. The specific operation methods of each module and unit have been described in detail in the method embodiments and will not be repeated here. In practical applications, the deformed text detection device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0119] Please see Figure 4 , Figure 4This is a schematic diagram illustrating the structure of an electronic device according to an exemplary embodiment of this application. The electronic device 400 includes a memory 410 and a processor 420. The processor 420 executes program instructions stored in the memory 410 to implement the steps in any of the above embodiments of the deformed text detection method. In a specific implementation scenario, the electronic device 400 may include, but is not limited to, a microcomputer or a server. Furthermore, the electronic device 400 may also include mobile devices such as laptops and tablets, without limitation.

[0120] Specifically, processor 420 controls itself and memory 410 to implement the steps in any of the above-described deformed text detection method embodiments. Processor 420 can also be referred to as a Central Processing Unit (CPU). Processor 420 may be an integrated circuit chip with signal processing capabilities. Processor 420 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 420 can be implemented using integrated circuit chips.

[0121] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating the structure of a computer-readable storage medium in an exemplary embodiment of this application. The computer-readable storage medium 500 stores program instructions 510 that can be executed by a processor. The program instructions 510 are used to implement the steps in any of the above embodiments of the deformed text detection method.

[0122] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0123] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for detecting deformed text, characterized in that, The method includes: Extract image features from the image to be processed, wherein the image to be processed is an image acquired from a specified text carrier, and the specified text carrier is easily deformed under force; The image features are input into a control point detection network to identify the coordinates of edge control points, thereby obtaining an edge control point sequence; wherein, the edge control point refers to the point on the edge of the text region in the image to be processed, and the edge control point sequence is represented as a two-dimensional matrix; The edge control point sequence is input into a differentiable B-spline layer to perform matrix multiplication on the basis function matrix and the edge control point sequence to obtain a B-spline curve point sequence; wherein, the basis function matrix refers to a two-dimensional matrix obtained by combining the B-spline basis function values; A text detection box is generated based on the B-spline curve point sequence.

2. The method according to claim 1, characterized in that, The method further includes: Calculate the model training loss based on the B-spline curve point sequence; The gradient of the model training loss with respect to the B-spline curve point sequence is calculated, and the gradient is backpropagated through the differentiable B-spline layer to update the control point detection network.

3. The method according to claim 2, characterized in that, The edge control point sequence includes an upper edge point sequence and a lower edge point sequence. The upper edge point sequence and the lower edge point sequence are respectively input into the differentiable B-spline layer, and the resulting B-spline curve point sequence includes an upper edge curve point sequence and a lower edge curve point sequence. The calculation of model training loss based on the B-spline curve point sequence includes: Based on the upper edge curve point sequence, the curvature of the upper edge curve at each sampling point is calculated to obtain the upper edge curvature; based on the lower edge curve point sequence, the curvature of the lower edge curve at each sampling point is calculated to obtain the lower edge curvature; based on the upper edge curvature and the lower edge curvature, the curvature smoothing loss is calculated. And / or, based on the spacing between the points of the upper edge curve point sequence and the lower edge curve point sequence, calculate the spacing change rate between the upper edge curve and the lower edge curve; based on the spacing change rate, calculate the edge parallelism loss; The model training loss is calculated based on the curvature smoothing loss and / or the edge parallelism loss.

4. The method according to claim 1, characterized in that, The control point detection network includes convolutional layers, activation functions, upsampling layers, and location extraction layers; the step of inputting the image features into the control point detection network to identify the coordinates of edge control points and obtain an edge control point sequence includes: The image features are input into the convolutional layer to obtain the convolutional features output by the convolutional layer. The number of channels in the convolutional features is equal to the number of edge control points to be identified. The convolutional features are input into the activation function to obtain the activation features output by the activation function, where the values ​​in the activation features represent the normalized coordinates. The activation features are upsampled based on the upsampling layer to obtain upsampled features; The upsampled features are input into the location extraction layer to determine the maximum response coordinates of each channel, thereby obtaining the edge control point sequence output by the location extraction layer.

5. The method according to claim 1, characterized in that, The method further includes: Get the preset clamped node vector; All B-spline basis function values ​​are recursively calculated based on the clamped node vectors. Organize all B-spline basis function values ​​into a two-dimensional matrix to obtain the basis function matrix.

6. The method according to claim 1, characterized in that, The edge control point sequence includes an upper edge point sequence and a lower edge point sequence. The upper edge point sequence and the lower edge point sequence are respectively input into the differentiable B-spline layer, and the resulting B-spline curve point sequence includes an upper edge curve point sequence and a lower edge curve point sequence. The process of generating a text detection box based on the B-spline curve point sequence includes: Based on the upper edge curve point sequence and the lower edge curve point sequence, calculate the short sides on both sides of the upper edge curve and the lower edge curve; By combining the upper edge curve, the lower edge curve, and the short side, a text detection box is generated.

7. The method according to claim 6, characterized in that, The starting point in the B-spline curve point sequence is equal to the starting edge control point in the edge control point sequence, and the ending point in the B-spline curve point sequence is equal to the ending edge control point in the edge control point sequence. The step of calculating the short sides on both sides of the upper and lower edge curves based on the upper edge curve point sequence and the lower edge curve point sequence includes: Calculate the tangent vectors corresponding to the starting point of the upper edge curve point sequence and the starting point of the lower edge curve point sequence to obtain the first tangent vector; calculate the tangent vectors corresponding to the ending point of the upper edge curve point sequence and the ending point of the lower edge curve point sequence to obtain the second tangent vector. Based on the first tangent vector and the second tangent vector, the short sides on both sides of the upper edge curve and the lower edge curve are generated respectively.

8. The method according to claim 1, characterized in that, The method further includes: The text content in the image to be processed is corrected based on the text detection box to obtain the corrected image to be processed. Text recognition is performed on the corrected image to obtain the text recognition result.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.

Images