A processing method, device and equipment for identifying a distorted text image and a medium

Abstract

The application provides a processing method, device and equipment for identifying a distorted text image and a medium, comprising: acquiring pixel data of a distorted surface image containing a text block, the pixel data comprising a plurality of pixel point data; inputting the pixel data into a semantic segmentation network to acquire pixel point data containing the text block in the distorted surface image; calculating a depth value of the pixel point data containing the text block, and generating a plurality of three-dimensional point data based on the depth value; clustering the three-dimensional point data based on normal vector data of the three-dimensional point data to generate a plurality of cluster data; and performing smoothing processing on initial coordinate data of the cluster data to generate intermediate coordinate data. Through the processing method, device and equipment for identifying a distorted text image and the medium, the distorted text image can be corrected and identified.

Description

Technical Field

[0001] This invention relates to the field of text image processing technology, and in particular to a processing method, apparatus, device, and medium for recognizing distorted text images. Background Technology

[0002] With the development of mobile camera devices, end users are increasingly adopting digital recording methods to save text images. For example, users commonly use mobile camera devices to take photos on the go, achieving efficient document recording. However, due to the instability of mobile camera devices, text images often contain various deformations, i.e., distorted surfaces. These distorted surfaces significantly interfere with subsequent text recognition processes. Therefore, how to effectively correct and recognize text images with distorted surfaces has become an urgent problem to be solved. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a processing method, apparatus, device and medium for recognizing distorted text images. This invention is capable of correcting and recognizing text images on distorted surfaces.

[0004] To achieve the above and other related objectives, the present invention provides a processing method, apparatus, device, and medium for recognizing distorted text images, comprising:

[0005] Obtain pixel data of a distorted surface image containing text blocks, wherein the pixel data includes multiple pixel data points;

[0006] The pixel data is input into a semantic segmentation network to obtain pixel data containing the text block in the distorted surface image;

[0007] Calculate the depth value of the pixel data containing the text block, and generate multiple three-dimensional point data based on the depth value;

[0008] Based on the normal vector data of the three-dimensional point data, the three-dimensional point data is clustered to generate multiple cluster data;

[0009] The initial coordinate data of the cluster data is smoothed to generate intermediate coordinate data;

[0010] The intermediate coordinate data is corrected based on the normal vector data of the cluster data to generate the target coordinate data.

[0011] The target coordinate data is projected onto a two-dimensional plane to generate corrected image data;

[0012] Text information in the corrected image data is identified, and text data is generated.

[0013] In one embodiment of the present invention, the step of inputting the pixel data into a semantic segmentation network to obtain pixel data containing the text block in the distorted surface image includes:

[0014] The pixel data is input into a semantic segmentation network, which outputs category probability data for each pixel. The category probability data is represented as... ,in, This indicates that a pixel's data corresponds to a certain category. The probability, This represents background pixel data or pixel data containing text blocks. This represents the weight vector corresponding to a specific pixel's data and its class j. Indicates a category index;

[0015] Based on the category probability data, obtain the pixel data containing the text block.

[0016] In one embodiment of the present invention, the step of clustering the three-dimensional point data based on the normal vector data of the three-dimensional point data to generate multiple cluster data includes:

[0017] Based on the preset neighborhood data of the three-dimensional point data, calculate the normal vector data of the three-dimensional point data;

[0018] Based on the normal vector data of the three-dimensional point data, calculate the neighbor point density data of each three-dimensional point data;

[0019] Based on the neighbor point density data, the three-dimensional point data is clustered to generate multiple cluster data.

[0020] In one embodiment of the present invention, the step of smoothing the initial coordinate data of the cluster data to generate intermediate coordinate data includes:

[0021] Based on the cluster data, a triangular grid of the cluster data is established;

[0022] The initial coordinate data of the cluster data in the triangular mesh is smoothed to generate intermediate coordinate data, which is represented as follows: ,in, Indicates the iteration of some initial coordinate data Intermediate coordinate data of the second time, This represents the set of neighboring points of a given 3D point in the cluster data. This represents the number of neighboring points of a given 3D point in the cluster data. Indicates the weight value. Indicates the number of iterations.

[0023] In one embodiment of the present invention, the step of correcting the intermediate coordinate data based on the normal vector data of the cluster data to generate the target coordinate data includes:

[0024] Obtain the normal vector data of the core point data in each of the cluster data;

[0025] Based on the normal vector data of the core point data, the intermediate coordinate data is corrected to generate the target coordinate data.

[0026] In one embodiment of the present invention, the step of correcting the intermediate coordinate data based on the normal vector data of the triangular mesh to generate the target coordinate data includes:

[0027] Based on the normal vector data of the core point data, the intermediate coordinate data is corrected.

[0028] The intermediate coordinate data after correction is transformed to generate the target coordinate data.

[0029] In one embodiment of the present invention, the step of projecting the target coordinate data onto a two-dimensional plane to generate corrected image data includes:

[0030] Eliminate the depth value from the target coordinate data to generate two-dimensional coordinate data;

[0031] The two-dimensional coordinate data is projected onto a two-dimensional plane to generate corrected image data.

[0032] The present invention also proposes a processing device for recognizing distorted text images, comprising:

[0033] The acquisition module is used to acquire pixel data of a distorted surface image containing text blocks, wherein the pixel data includes multiple pixel data points;

[0034] The segmentation module is used to input the pixel data into the semantic segmentation network to obtain pixel data containing the text block in the distorted surface image;

[0035] The generation module is used to calculate the depth value of the pixel data containing the text block, and generate multiple three-dimensional point data based on the depth value;

[0036] The clustering module is used to cluster the three-dimensional point data based on the normal vector data of the three-dimensional point data, and generate multiple cluster data.

[0037] A smoothing module is used to smooth the initial coordinate data of the cluster data to generate intermediate coordinate data.

[0038] The correction module is used to correct the intermediate coordinate data based on the normal vector data of the cluster data to generate target coordinate data;

[0039] The projection module is used to project the target coordinate data onto a two-dimensional plane to generate corrected image data;

[0040] The recognition module is used to recognize the text information in the corrected image data and generate text data.

[0041] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a processing method for recognizing distorted text images.

[0042] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a processing method for recognizing distorted text images.

[0043] As described above, the present invention provides a processing method, apparatus, device and medium for recognizing distorted text images, which can use the geometric information of three-dimensional objects in the image to perform text correction and recognition, improve the image correction effect, and the correction process does not require the support of specific equipment. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of an application environment for a method for recognizing distorted text images according to an embodiment of the present invention;

[0046] Figure 2 This is a flowchart illustrating a method for recognizing distorted text images according to an embodiment of the present invention;

[0047] Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S20;

[0048] Figure 4 yes Figure 2 A schematic diagram of a specific implementation method for step S40;

[0049] Figure 5 yes Figure 2A schematic diagram of a specific implementation method for step S50;

[0050] Figure 6 yes Figure 2 A schematic diagram of a specific implementation method for step S60;

[0051] Figure 7 yes Figure 6 A schematic diagram of a specific implementation method for step S62;

[0052] Figure 8 yes Figure 2 A schematic diagram of a specific implementation of step S70;

[0053] Figure 9 This is a schematic diagram of a processing device for recognizing distorted text images according to an embodiment of the present invention.

[0054] Figure 10 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0055] Figure 11 This is another structural schematic diagram of a computer device according to one embodiment of the present invention.

[0056] Component designation explanation:

[0057] 101. Acquisition Module; 102. Segmentation Module; 103. Generation Module; 104. Clustering Module; 105. Smoothing Module; 106. Correction Module; 107. Projection Module; 108. Recognition Module. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] The present invention provides a method for identifying distorted text images, which can be applied to applications such as... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain pixel data of a distorted surface image containing text blocks from the client. This pixel data includes multiple pixel points. The pixel data is input into a semantic segmentation network to obtain pixel data containing the text blocks within the distorted surface image. The depth value of the pixel data containing the text blocks is calculated, and multiple 3D point data are generated based on the depth value. The 3D point data is clustered based on the normal vector data of the 3D point data to generate multiple cluster data. The initial coordinate data of the cluster data is smoothed to generate intermediate coordinate data. The intermediate coordinate data is corrected based on the normal vector data of the cluster data to generate target coordinate data. The target coordinate data is projected onto a 2D plane to generate corrected image data. The text information in the corrected image data is recognized to generate text data. In this invention, text recognition is performed using the geometric information of 3D objects in an image, without the need for specific device support. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The present invention will now be described in detail through specific embodiments.

[0060] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a method for recognizing distorted text images provided in an embodiment of the present invention includes the following steps:

[0061] Step S10: Obtain pixel data of a distorted surface image containing text blocks, wherein the pixel data includes multiple pixel data points.

[0062] In one embodiment, the intelligent question-answering processing method provided by the present invention can be applied to various text image recognition scenarios. Text image recognition is typically implemented through a server, which can receive pixel data of text images from a client. Specifically, the distorted surface image containing text blocks can be a wrinkled document, a product casing with creases, or other text images with distorted surfaces. The client can store distorted surface image data containing text blocks, which can be composed of multiple pixels. This image data is the pixel data of the image, and the pixel data can include multiple pixel data. During text image correction and recognition, the client can transmit the pixel data of the distorted surface image containing text blocks to the server for correction and recognition operations.

[0063] It should be noted that the multiple pixel data may include background pixel data and pixel data containing text blocks.

[0064] Step S20: Input the pixel data into the semantic segmentation network to obtain the pixel data of the text block contained in the distorted surface image.

[0065] In one embodiment, since the multiple pixel data may include background pixel data and pixel data containing text blocks, the background pixel data can interfere with the text image correction and recognition process. Therefore, the server can extract the pixel data containing text blocks from the pixel data to perform correction and recognition on the pixel data containing text blocks separately, thereby eliminating the interference of background pixel data.

[0066] Please see Figure 3 , Figure 3 for Figure 2 A schematic flowchart illustrating a specific implementation of step S20. Step S20, which involves inputting the pixel data into a semantic segmentation network to obtain pixel data containing the text block in the distorted surface image, may include:

[0067] Step S21: Input the pixel data into the semantic segmentation network and output the category probability data for each pixel data point, wherein the category probability data is represented as... ,in, This indicates that a pixel's data corresponds to a certain category. The probability, This represents background pixel data or pixel data containing text blocks. This represents the weight vector corresponding to a specific pixel's data and its class j. This indicates a category index.

[0068] Step S22: Based on the category probability data, obtain the pixel data containing the text block.

[0069] For steps S21-S22, when acquiring pixel data containing text blocks, the pixel data of the distorted surface image containing text blocks can be input into the semantic segmentation network to segment out the pixel data containing text blocks. Here, the semantic segmentation network can be a semantic segmentation model based on a convolutional neural network.

[0070] Specifically, the pixel data of the distorted surface image containing text blocks can be used as parameters and input into the convolutional neural network. That is, each pixel data point can be used as an input value into the convolutional neural network. Each input pixel data point corresponds to two output nodes in the convolutional neural network. One output node can be a weight vector corresponding to background class pixels for a given pixel data point, and the other output node can be a weight vector corresponding to pixels containing text blocks for a given pixel data point. The larger the weight vector corresponding to a certain class of pixels for a given pixel data point, the greater the probability that the pixel data point belongs to that class of pixels.

[0071] Since weight vectors cannot intuitively represent the probability of each pixel corresponding to a certain category of pixel data, a classification probability formula can be introduced in the last layer of the convolutional neural network. Based on the weight vector of each pixel data, the classification probability formula can be used to calculate the category probability data of each pixel data corresponding to a certain category of pixel data. This allows the classification of each pixel data to be determined. Specifically, the category probability data can be represented as... ,in, This indicates that a pixel's data corresponds to a certain category. The probability, This represents background pixel data or pixel data containing text blocks. This represents the weight vector corresponding to a specific pixel's data and its class j. This indicates a category index.

[0072] For example, the weight vector corresponding to a certain pixel data point in the background class pixel data is: If the weight vector j of a certain pixel data point corresponding to the pixel data point containing the text block is j, then the category probability data of this pixel data point corresponding to the background class pixel data is j. The category probability data for this pixel data corresponding to the pixel data containing the text block is: .like If so, then this pixel data belongs to the background pixel category. If the probability data indicates a text block, then this pixel data is the pixel data containing the text block. Therefore, based on this category probability data, background pixel data and pixel data containing text blocks can be identified, thereby segmenting the pixel data containing text blocks.

[0073] Step S30: Calculate the depth value of the pixel data containing the text block, and generate multiple three-dimensional point data based on the depth value.

[0074] In one embodiment, the pixel data containing the text block can be converted into three-dimensional point data to utilize three-dimensional information in the image for correction.

[0075] For example, the coordinate data of a certain pixel point can be: This coordinate data is two-dimensional coordinate data, meaning it has coordinate values ​​on the X and Y axes. Based on this two-dimensional coordinate data, its Z-axis coordinate value can be obtained, which can be represented as... Therefore, the three-dimensional coordinate data of a specific pixel can be obtained. It should be noted that the coordinates of a pixel on the Z-axis are its depth values, which can refer to the distance of the pixel from the shooting device.

[0076] It is important to note that when calculating the depth value of each pixel, the encoder can first be used to extract the two-dimensional coordinate data of each pixel. The two-dimensional coordinate data can be represented as follows: Next, the decoder analyzes the two-dimensional coordinate data and obtains the depth value of each pixel. Specifically, the depth value can be represented as... ,in, It is two-dimensional coordinate data. This is a mapping function. In summary, each pixel data can be transformed into three-dimensional point data, and the coordinate data of each three-dimensional point can be represented as (…). Multiple 3D point data can form a point cloud set.

[0077] Step S40: Cluster the three-dimensional point data based on the normal vector data of the three-dimensional point data to generate multiple cluster data.

[0078] In one embodiment, it should be noted that each 3D point data has a normal vector data, which can be the normal vector of the fitting surface formed by each 3D point data and its neighboring points. Neighboring points can refer to other 3D points in the neighborhood of each 3D point data.

[0079] It is important to note that the neighborhood of a given 3D point data point refers to any open interval centered on that 3D point data point. For example, if a 3D point data point can be denoted as 'a', and the radius of its corresponding neighborhood can be denoted as 'δ', then the open interval of the 3D point data point can be represented as (a-δ, a+δ). Other 3D point data points within this open interval are the neighboring point data points of this 3D point data point.

[0080] Please see Figure 4 , Figure 4 for Figure 2 A schematic flowchart illustrating a specific implementation of step S40. Step S40, which involves clustering the 3D point data based on the normal vector data of the 3D point data to generate multiple cluster data, may include:

[0081] Step S41: Calculate the normal vector data of the three-dimensional point data based on the preset neighborhood data of the three-dimensional point data;

[0082] Step S42: Calculate the neighbor density data of each three-dimensional point data based on the normal vector data of the three-dimensional point data;

[0083] Step S43: Based on the neighbor point density data, cluster the three-dimensional point data to generate multiple cluster data.

[0084] For steps S41-S43, the preset neighborhood data may include the radius parameter of the neighborhood. And the minimum number of neighboring points parameter. The radius parameter based on each 3D point data and its neighborhood. The range of each neighborhood can be determined, and the center point of each neighborhood is the corresponding three-dimensional point data.

[0085] It's important to note that calculating the normal vector of a given 3D point is equivalent to calculating the normal vector of the plane formed by that point and its neighbors. This normal vector can be described as the point along which the projections of the given point and its neighbors are most densely distributed. Therefore, to calculate the normal vector of a given 3D point, we can first establish the covariance matrix of the fitted plane for each point and its neighbors using principal component analysis. The smallest eigenvalue and its corresponding eigenvector of the covariance matrix indicate that the variance of the given point and its neighbors is minimized and their distribution is most dense along the direction of this eigenvector. Therefore, the eigenvector corresponding to the smallest eigenvalue of the covariance matrix is ​​the normal vector to be calculated.

[0086] Specifically, the covariance matrix of the fitting plane of a certain 3D point data and its neighboring points can be expressed as: .in, Indicates the radius of the neighborhood Distance within the range Recent 3D point data, Indicates the radius of the neighborhood Distance within the range Recent The number of points The neighborhood center data of a certain 3D point data. The value can be set according to the calculation precision. This covariance matrix The eigenvector corresponding to the smallest eigenvalue is the normal vector to be calculated.

[0087] In one embodiment, when performing clustering operations on 3D point data, multiple 3D point data are first set as core points, and the corresponding neighboring points are clustered around the core points to obtain multiple clusters. It should be noted that each core point can be determined based on the normal vector data of each 3D point data.

[0088] Specifically, a random 3D point can be selected and marked as a visited point. The neighborhood radius parameter of this 3D point is set to... The angle difference between the normal vector of the 3D point data and the normal vector of each 3D point data in its neighborhood is calculated. Based on this angle difference, the neighbor density data of the 3D point data can be calculated. The neighbor density data is the number of points in the neighborhood whose angle difference is less than a threshold value. If the neighbor density data is greater than the minimum number of neighbor points parameter, the 3D point data can be set as the core point, and this core point can be set as a cluster. A cluster This can represent information about a distorted surface. Finally, points in this neighborhood smaller than the difference threshold can be clustered into clusters. In, and update the cluster .

[0089] It should be noted that this can be done for each cluster. Each 3D point in the dataset is visited once, and the visited 3D point is marked as a visited point. Similarly, the neighbor density data of each 3D point data can be calculated, and each cluster can be... The 3D point with the highest nearest neighbor density is updated as the core point. Simultaneously, points within the neighborhood of each core point whose angular difference in the normal vector is less than a threshold are clustered into each cluster. middle.

[0090] It is important to note that after visiting each cluster... After collecting all 3D point data, unclustered 3D points are marked as noise points and removed.

[0091] Step S50: Smooth the initial coordinate data of the cluster data to generate intermediate coordinate data.

[0092] In one embodiment, the initial coordinate data of each three-dimensional point in the cluster can be smoothed to adjust each initial coordinate data to be approximately the coordinate data of its neighboring points, thus completing the initial adjustment of the distorted surface information.

[0093] Please see Figure 5 , Figure 5 for Figure 2 A schematic flowchart illustrating a specific implementation of step S50. Step S50, which smooths the initial coordinate data of the cluster data to generate intermediate coordinate data, may include:

[0094] Step S51: Based on the cluster data, establish a triangular mesh of the cluster data;

[0095] Step S52: Smooth the initial coordinate data of the cluster data in the triangular mesh and generate intermediate coordinate data, which is represented as follows:

[0096]

[0097] in, Indicates the iteration of some initial coordinate data Intermediate coordinate data of the second time, This represents the set of neighboring points of a given 3D point in the cluster data. This represents the number of neighboring points of a given 3D point in the cluster data. Indicates the weight value. Indicates the number of iterations. Indicates adjacent points Coordinate data.

[0098] For steps S51-S52, the spatial region growing algorithm based on triangulation can establish a triangular topological structure for each cluster data, forming a triangular mesh of multiple 3D point data. Specifically, three 3D points from a certain cluster data are selected to form an initial triangle, and then another 3D point data is selected and connected to one edge of the initial triangle to form a new triangle. The same steps are repeated until a triangular mesh of all 3D point data is formed.

[0099] Specifically, during the smoothing operation, the initial coordinate data of each 3D point can be represented as 3D coordinates in the Cartesian coordinate system. Based on this initial coordinate data, the offset between the center of the corresponding 3D point data and the center of its adjacent 3D point data can be obtained. This offset can be expressed as a 3D coordinate in the Laplace coordinate system. Laplace coordinates It can be represented as .in This represents the set of neighboring points of a given 3D point in the cluster data. This represents the number of neighboring points of a specific 3D point in the cluster data.

[0100] Based on the three-dimensional coordinates in this Laplace coordinate system Iterative smoothing operations can be performed to obtain intermediate coordinate data. Specifically, the initial Laplace coordinates can be... The obtained coordinate data after the first smoothing is = . This represents the weight value, which can be set to 0.1 or other values ​​depending on the calculation requirements. The above summarizes the first iteration; similarly, a second iteration can be performed. This is based on coordinate data. The Laplace coordinates can be calculated a second time. Laplace coordinates It can be represented as The obtained coordinate data after the second smoothing for = This is the second iteration. Following the same steps, multiple iterations can be performed. The number of iterations is unlimited and can be set according to computational needs. Based on the above iterative smoothing process, the final intermediate coordinate data can be obtained. The intermediate coordinate data is in the Cartesian coordinate system.

[0101] Step S60: Correct the intermediate coordinate data based on the normal vector data of the cluster data to generate target coordinate data.

[0102] In one embodiment, the core point data and corresponding normal vector data of the core points in steps S41-S43 can be obtained. Based on the normal vector of each core point, the intermediate coordinates of other 3D points in the cluster corresponding to each core point can be corrected. This allows the intermediate coordinates of the remaining 3D points to be projected onto the plane corresponding to the core point, ensuring that all 3D points in each cluster are in the same plane after correction. Finally, coordinate transformation is performed on the intermediate coordinate data of each corrected 3D point to obtain the target coordinate data, ensuring that the depth values ​​of the coordinate data of each 3D point remain consistent. In the above steps, the geometric information of the 3D points is fully utilized for correction.

[0103] Please see Figure 6 , Figure 6 for Figure 1 A schematic flowchart illustrating a specific implementation of step S60. Step S60, which involves correcting the intermediate coordinate data based on the normal vector data of the cluster data to generate the target coordinate data, may include:

[0104] S61. Obtain the normal vector data of the core point data in each cluster data;

[0105] S62. Based on the normal vector data of the core point data, the intermediate coordinate data is corrected to generate target coordinate data.

[0106] For steps S61-S62, based on the normal vector data of the core points, the intermediate coordinate data can be corrected. For example, each core point and its normal vector can define a plane. According to the equation of this plane, the coordinates of each 3D point in each cluster can be corrected so that the orientation of the normal vector of the mesh surface where each 3D point in each cluster is located is consistent with the orientation of the normal vector of the corresponding core point. Thus, the mesh surface where each 3D point in each cluster is located has the same orientation, and the distorted surface represented by each cluster can be transformed into a plane.

[0107] Specifically, when correcting the coordinates of each 3D point in each cluster, the intermediate coordinate data of a core point can be represented as: The corrected coordinates can be expressed as The normal vector corresponding to this core point can be expressed as: A plane can be determined by the core point and its normal vector, and the equation of this plane can be expressed as: The corrected coordinates can be represented as follows: , .

[0108] Please see Figure 7 , Figure 7 for Figure 6 A schematic flowchart illustrating a specific implementation of step S62. Step S62, which involves correcting the intermediate coordinate data based on the normal vector data of the core point data to generate the target coordinate data, may include:

[0109] Step S621: Based on the normal vector data of the core point data, perform correction processing on the intermediate coordinate data;

[0110] Step S622: Perform coordinate transformation on the intermediate coordinate data after correction to generate target coordinate data.

[0111] For steps S621-S622, based on the normal vector data of each core point, coordinate transformation can be performed on the corrected intermediate coordinates to ensure that the depth values ​​of each intermediate coordinate are consistent. When the depth values ​​of each intermediate coordinate are consistent, the tilted plane represented by each cluster can be transformed from a tilted state to a vertical state. Therefore, when the coordinate data of each cluster is subsequently projected onto a two-dimensional plane, image distortion can be reduced.

[0112] Specifically, the normal vector corresponding to each core point can be represented as: When performing coordinate transformations, , , , .in, sin , , ,in, , , These can be coordinate matrices that rotate about the x, y, and z axes, respectively. This can be the final three-dimensional transformation matrix. Based on this three-dimensional transformation matrix, the coordinate matrix formed by the coordinates of each three-dimensional point in the Cartesian coordinate system can be obtained. A rotation transformation is performed to ensure that the depth values ​​of each 3D point coordinate remain consistent. Specifically, the target coordinate matrix formed by the coordinates of all 3D points can be represented as follows: .in, This is the final three-dimensional transformation matrix described above.

[0113] Step S70: Project the target coordinate data onto a two-dimensional plane to generate corrected image data.

[0114] In one embodiment, the target coordinate data of each three-dimensional point in each cluster can be obtained based on the target coordinate matrix formed by the coordinates of each three-dimensional point.

[0115] It should be noted that before projecting the target coordinate data onto a two-dimensional plane, the depth value, i.e., the coordinate value of the target coordinate data on the Z-axis, needs to be removed from the target coordinate data.

[0116] Please see Figure 8 , Figure 8 for Figure 2 A schematic flowchart illustrating a specific implementation of step S70. Step S70, which projects the target coordinate data onto a two-dimensional plane to generate corrected image data, may include:

[0117] S71. Eliminate the depth value of the target coordinate data to generate two-dimensional coordinate data;

[0118] S72. Project the two-dimensional coordinate data onto a two-dimensional plane to generate corrected image data.

[0119] For steps S71-S72, the target coordinate data of a certain three-dimensional point can be represented as follows: After eliminating depth values, the target coordinate data of a 3D point can be represented as: .

[0120] It's important to note that projecting each 3D point (after removing depth values) onto a 2D plane will result in missing pixels. These missing pixels can be filled using bilinear interpolation. Specifically, the pixel value of a 2D point in the 2D plane is unknown. Since the pixel value of a missing pixel is similar to the pixel value of a point in the original coordinate system, each missing pixel can correspond to a certain position in the original coordinate system. The missing pixel value can be calculated using the pixel values ​​of the four points adjacent to that position in the original coordinate system. For example, the coordinates of the four adjacent points can be represented as Q11 (x1, y1), Q12 (x1, y2), Q21 (x2, y1), and Q22 (x2, y2). The pixel values ​​of the four adjacent points can be represented as f(Q11), f(Q12), f(Q21), and f(Q22). The missing pixel value of a two-dimensional point in the two-dimensional plane can be represented as f(x, y) = f(Q11) / [(x2-x1)(y2-y1)](x2-x)(y2-y) + f(Q21) / [(x2-x1)(y2-y1)](x-x1)(y2-y) + f(Q12) / [(x2-x1)(y2-y1)](x2-x)(y-y1) + f(Q22) / [(x2-x1)(y2-y1)](x-x1)(y-y1). After filling in the missing pixel values ​​of each two-dimensional point in the two-dimensional plane, corrected image data can be generated.

[0121] Step S80: Identify the text information in the corrected image data and generate text data.

[0122] In one embodiment, when recognizing text information from corrected image data, a convolutional neural network can be used to obtain the final text content. Other text recognition methods, such as long short-term memory networks, can also be used.

[0123] Specifically, taking the use of convolutional neural networks for recognition as an example, the corrected image data can be input into a text detection network with a feature pyramid network and a fully convolutional neural network as its core, outputting a text centerline score map and a text region score map. The text centerline score map reflects the position of the text centerline. The text centerline score map reflects the position of the text region boundaries. The text region score map is then masked using the text centerline score map to obtain a masked text region score map. The masked text region score map reflects the position of the text region. Mapping the masked text region score map to the corresponding corrected image data yields the text region. The text region is then cropped into a single line of text and input into a text recognition model with a Transformer neural network as its main architecture. To simplify the encoding process, the model uses a convolutional neural network with a residual network as its main architecture instead of an encoder. Locality-sensitive hashing (LSH) is used instead of traditional embedding to reduce model dimensionality and improve iteration speed. Finally, the data is decoded character by character to generate text, forming single-line text data.

[0124] As can be seen, the above scheme can utilize the geometric information of three-dimensional objects in the image to perform text correction and recognition, improve the image correction effect, and the correction process does not require the support of specific equipment.

[0125] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0126] In one embodiment, a processing apparatus for recognizing distorted text images is provided, which corresponds one-to-one with the processing method for recognizing distorted text images in the above embodiments. For example... Figure 9 As shown, the processing device for recognizing distorted text images includes an acquisition module 101, a segmentation module 102, a generation module 103, a clustering module 104, a smoothing module 105, a correction module 106, a projection module 107, and a recognition module 108. Detailed descriptions of each functional module are as follows:

[0127] The acquisition module 101 is used to acquire pixel data of a distorted surface image containing text blocks, wherein the pixel data includes multiple pixel data.

[0128] The segmentation module 102 is used to input the pixel data into the semantic segmentation network to obtain pixel data containing the text block in the distorted surface image;

[0129] The generation module 103 is used to calculate the depth value of the pixel data containing the text block, and generate multiple three-dimensional point data based on the depth value;

[0130] Clustering module 104 is used to cluster the three-dimensional point data based on the normal vector data of the three-dimensional point data to generate multiple cluster data.

[0131] Smoothing module 105 is used to smooth the initial coordinate data of the cluster data to generate intermediate coordinate data;

[0132] Correction module 106 is used to correct the intermediate coordinate data based on the normal vector data of the cluster data to generate target coordinate data;

[0133] Projection module 107 is used to project the target coordinate data onto a two-dimensional plane to generate corrected image data;

[0134] The recognition module 108 is used to recognize the text information in the corrected image data and generate text data.

[0135] In one embodiment, the segmentation module 102 is specifically used for:

[0136] The pixel data is input into a semantic segmentation network, which outputs category probability data for each pixel. The category probability data is represented as... ,in, This indicates that a pixel's data corresponds to a certain category. The probability, This represents background pixel data or pixel data containing text blocks. This represents the weight vector corresponding to a specific pixel's data and its class j. Indicates a category index;

[0137] Based on the category probability data, obtain the pixel data containing the text block.

[0138] In one embodiment, the clustering module 104 is specifically used for:

[0139] Based on the preset neighborhood data of the three-dimensional point data, calculate the normal vector data of the three-dimensional point data;

[0140] Based on the normal vector data of the three-dimensional point data, calculate the neighbor point density data of each three-dimensional point data;

[0141] Based on the neighbor point density data, the three-dimensional point data is clustered to generate multiple cluster data.

[0142] In one embodiment, the smoothing module 105 is specifically used for:

[0143] Based on the cluster data, a triangular grid of the cluster data is established;

[0144] The initial coordinate data of the cluster data in the triangular mesh is smoothed to generate intermediate coordinate data, which is represented as follows: ,in, Indicates the iteration of some initial coordinate data Intermediate coordinate data of the second time, This represents the set of neighboring points of a given 3D point in the cluster data. This represents the number of neighboring points of a given 3D point in the cluster data. Indicates the weight value. Indicates the number of iterations.

[0145] In one embodiment, the correction module 106 is specifically used for:

[0146] Obtain the normal vector data of the core point data in each of the cluster data;

[0147] Based on the normal vector data of the core point data, the intermediate coordinate data is corrected to generate the target coordinate data.

[0148] In one embodiment, the correction module 106 is further configured to:

[0149] Based on the normal vector data of the core point data, the intermediate coordinate data is corrected.

[0150] The intermediate coordinate data after correction is transformed to generate the target coordinate data.

[0151] In one embodiment, the projection module 107 is specifically used for:

[0152] Eliminate the depth value from the target coordinate data to generate two-dimensional coordinate data;

[0153] The two-dimensional coordinate data is projected onto a two-dimensional plane to generate corrected image data.

[0154] This invention provides a processing device for recognizing distorted text images, which can use the geometric information of three-dimensional objects in the image to perform text correction and recognition, improve the image correction effect, and the correction process does not require the support of specific equipment.

[0155] Specific limitations regarding the processing device for recognizing distorted text images can be found in the limitations of the processing method for recognizing distorted text images above, and will not be repeated here. Each module in the aforementioned intelligent question-answering processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0156] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side method for recognizing distorted text images.

[0157] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a processing method for recognizing distorted text images.

[0158] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0159] Obtain pixel data of a distorted surface image containing text blocks, wherein the pixel data includes multiple pixel data points;

[0160] The pixel data is input into a semantic segmentation network to obtain pixel data containing the text block in the distorted surface image;

[0161] Calculate the depth value of the pixel data containing the text block, and generate multiple three-dimensional point data based on the depth value;

[0162] Based on the normal vector data of the three-dimensional point data, the three-dimensional point data is clustered to generate multiple cluster data;

[0163] The initial coordinate data of the cluster data is smoothed to generate intermediate coordinate data;

[0164] The intermediate coordinate data is corrected based on the normal vector data of the cluster data to generate the target coordinate data.

[0165] The target coordinate data is projected onto a two-dimensional plane to generate corrected image data;

[0166] Text information in the corrected image data is identified, and text data is generated.

[0167] In one embodiment, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0168] Obtain pixel data of a distorted surface image containing text blocks, wherein the pixel data includes multiple pixel data points;

[0169] The pixel data is input into a semantic segmentation network to obtain pixel data containing the text block in the distorted surface image;

[0170] Calculate the depth value of the pixel data containing the text block, and generate multiple three-dimensional point data based on the depth value;

[0171] Based on the normal vector data of the three-dimensional point data, the three-dimensional point data is clustered to generate multiple cluster data;

[0172] The initial coordinate data of the cluster data is smoothed to generate intermediate coordinate data;

[0173] The intermediate coordinate data is corrected based on the normal vector data of the cluster data to generate the target coordinate data.

[0174] The target coordinate data is projected onto a two-dimensional plane to generate corrected image data;

[0175] Text information in the corrected image data is identified, and text data is generated.

[0176] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0177] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0179] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A processing method for recognizing distorted text images, characterized in that, include: Obtain pixel data of a distorted surface image containing text blocks, wherein the pixel data includes multiple pixel data points; The pixel data is input into a semantic segmentation network to obtain pixel data containing the text block in the distorted surface image; Calculate the depth value of the pixel data containing the text block, and generate multiple three-dimensional point data based on the depth value; Based on the normal vector data of the three-dimensional point data, the three-dimensional point data is clustered to generate multiple cluster data; Based on the cluster data, a triangular grid of the cluster data is established; The initial coordinate data of the cluster data in the triangular mesh are smoothed to generate intermediate coordinate data, which is represented as follows: ,in, Indicates the iteration of some initial coordinate data Intermediate coordinate data of the second time, This represents the set of neighboring points of a given 3D point in the cluster data. This represents the number of neighboring points of a given 3D point in the cluster data. Indicates the weight value. Indicates the number of iterations; The intermediate coordinate data is corrected based on the normal vector data of the cluster data to generate the target coordinate data. The target coordinate data is projected onto a two-dimensional plane to generate corrected image data; Identify text information in the corrected image data and generate text data. The step of clustering the three-dimensional point data based on the normal vector data of the three-dimensional point data to generate multiple cluster data includes: Based on the preset neighborhood data of the three-dimensional point data, calculate the normal vector data of the three-dimensional point data; Based on the normal vector data of the three-dimensional point data, calculate the neighbor point density data of each three-dimensional point data; Based on the neighbor point density data, the three-dimensional point data is clustered to generate multiple cluster data.

2. The processing method for recognizing distorted text images according to claim 1, characterized in that, The step of inputting the pixel data into a semantic segmentation network to obtain pixel data containing the text block in the distorted surface image includes: The pixel data is input into a semantic segmentation network, which outputs category probability data for each pixel. The category probability data is represented as... ,in, This indicates that a pixel's data corresponds to a certain category. The probability, This represents background pixel data or pixel data containing text blocks. This represents the weight vector corresponding to a specific pixel's data and its class j. Indicates a category index; Based on the category probability data, obtain the pixel data containing the text block.

3. The processing method for recognizing distorted text images according to claim 1, characterized in that, The step of correcting the intermediate coordinate data based on the normal vector data of the cluster data to generate the target coordinate data includes: Obtain the normal vector data of the core point data in each of the cluster data; Based on the normal vector data of the core point data, the intermediate coordinate data is corrected to generate the target coordinate data.

4. The processing method for recognizing distorted text images according to claim 3, characterized in that, The step of correcting the intermediate coordinate data based on the normal vector data of the core point data to generate the target coordinate data includes: Based on the normal vector data of the core point data, the intermediate coordinate data is corrected. The intermediate coordinate data after correction is transformed to generate the target coordinate data.

5. The processing method for recognizing distorted text images according to claim 1, characterized in that, The step of projecting the target coordinate data onto a two-dimensional plane to generate corrected image data includes: Eliminate the depth value from the target coordinate data to generate two-dimensional coordinate data; The two-dimensional coordinate data is projected onto a two-dimensional plane to generate corrected image data.

6. A processing apparatus for recognizing distorted text images, characterized in that, The processing apparatus comprising the processing method for recognizing distorted text images as described in any one of claims 1 to 5, wherein the processing apparatus includes: The acquisition module is used to acquire pixel data of a distorted surface image containing text blocks, wherein the pixel data includes multiple pixel data points; The segmentation module is used to input the pixel data into the semantic segmentation network to obtain pixel data containing the text block in the distorted surface image; A generation module is used to calculate the depth value of the pixel data containing the text block, and generate multiple three-dimensional point data based on the depth value; The clustering module is used to cluster the three-dimensional point data based on the normal vector data of the three-dimensional point data, and generate multiple cluster data. A smoothing module is used to smooth the initial coordinate data of the cluster data to generate intermediate coordinate data. The correction module is used to correct the intermediate coordinate data based on the normal vector data of the cluster data to generate target coordinate data; The projection module is used to project the target coordinate data onto a two-dimensional plane to generate corrected image data; The recognition module is used to recognize the text information in the corrected image data and generate text data.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the processing method for recognizing distorted text images as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the processing method for recognizing distorted text images as described in any one of claims 1 to 5.

Images