A text image direction correction method, device and electronic equipment

Abstract

The application belongs to the technical field of image processing, and provides a text image direction correction method, device and electronic equipment, the method comprising: acquiring a first text image; performing a rotation operation on the first text image according to N predetermined rotation angles to obtain N rotated images, wherein N is a natural number greater than or equal to two; estimating the correction direction of the first text image according to the N rotated images; and performing direction correction on the first text image according to the correction direction of the first text image. The application comprehensively judges the correction direction of the first text image according to the rotated images of multiple directions obtained after synchronous rotation of the first text image, has higher accuracy, can quickly identify the correction direction of the text image, can also identify the correction direction of the text image not containing text, has the advantages of simple operation, fast identification, wide application range, etc.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and is particularly suitable for image direction correction technology, and more particularly relates to a text image direction correction method and device, an electronic device and a computer readable medium.

[0002] The text image referred to herein refers to an image containing characters and / or patterns;

[0003] The text image direction referred to herein refers to the arrangement direction of the characters and / or patterns in the text image in line with reading habits, for example, the characters are upright, not skewed and not inverted.

[0004] The recognition direction referred to herein refers to the direction of the characters or patterns predetermined in the title recognition process, which is generally consistent with the usual arrangement direction of the text or image when reading. BACKGROUND

[0005] At present, there are more and more photograph searching products on the market. Through such products, users only need to take and upload the image of the title to be searched to find the same or similar title and answer, greatly facilitating the title search.

[0006] The existing photograph searching is mainly realized through image recognition and title searching. The characters or patterns in the text image uploaded by the user are first recognized, and then the existing title in the title database is searched quickly to find the same or similar title and answer. However, in practice, the direction of the characters or patterns in the text image taken by the user (i.e. the direction of the text image) is not always consistent with the recognition direction of the characters or patterns predetermined in the title recognition (i.e. the usual reading direction), and the characters or patterns may be inverted (or skewed more seriously) in the title recognition, which leads to incorrect recognition of the characters or patterns, affects the accuracy of the title search, and reduces the user experience. SUMMARY

[0007] (I) Technical problem to be solved

[0008] The present application aims to solve the technical problem that the direction of the text image taken is not consistent with the recognition direction, leading to incorrect title recognition.

[0009] (II) Technical scheme

[0010] To solve the above technical problem, one aspect of the present application proposes a text image direction correction method, which comprises the following steps:

[0011] Obtaining a first text image;

[0012] Rotating the first text image according to N predetermined rotation angles to obtain N rotated images, wherein N is a natural number greater than or equal to two.

[0013] The N rotated images are used to estimate a correction direction of the first text image, the correction direction being a direction after a rotation operation is performed on the first text image to facilitate subsequent image recognition;

[0014] The first text image is direction-corrected according to the correction direction of the first text image.

[0015] According to a preferred embodiment of the present application, the N rotated images are used to estimate a correction direction of the first text image, including:

[0016] The N rotated images are input into a trained direction estimation model to estimate a probability of a direction of each rotated image being correct, and a direction of a rotated image with the highest probability is taken as the correction direction of the first text image.

[0017] Optionally, the direction estimation model is generated after an image classification model is trained.

[0018] According to a preferred embodiment of the present application, the direction estimation model includes N neural networks and a judgment module connected to the N neural networks respectively, the N neural networks respectively calculating a probability of a direction of an input rotated image being correct, and the judgment module judging the correction direction of the first text image according to a probability level of a direction of each rotated image being correct.

[0019] Optionally, the neural network includes a shallow convolutional network and a self-attention neural network connected in sequence, the self-attention neural network being connected to the judgment module, the shallow convolutional network being used to extract feature data of a rotated image, and the self-attention neural network being used to obtain the probability of the direction of the rotated image being correct according to the feature data.

[0020] Optionally, the shallow convolutional network includes a plurality of convolutional blocks and a fully connected layer.

[0021] Optionally, each convolutional block includes a convolutional layer, a pooling layer, a batch normalization layer and an excitation layer.

[0022] According to a preferred embodiment of the present application, the self-attention neural network is based on a Transformer model.

[0023] Optionally, the self-attention neural network includes a self-attention module and a binary classification module, the self-attention module being used to convert an input image into a specification meeting a requirement of the binary classification module, and the binary classification module being used to obtain the probability of the direction of the rotated image being correct.

[0024] Optionally, the binary classification module includes a fully connected layer and a softmax layer.

[0025] According to a preferred embodiment of the present application, the rotating operation is performed on the first text image according to N predetermined rotation angles to obtain N rotated images, including:

[0026] The first text image is converted into a first text image matrix;

[0027] The first text image matrix is subjected to matrix operation to obtain N rotation matrices corresponding to the N rotated images.

[0028] According to a preferred embodiment of the present application, the predetermined rotation angles are four, including: no rotation of the first text image, rotating the first text image by 90 degrees along a first direction, rotating the first text image by 180 degrees along the first direction, and rotating the first text image by 270 degrees along the first direction.

[0029] The first text image matrix is subjected to matrix operation to obtain N rotation matrices corresponding to the N rotated images, including:

[0030] The first text image matrix is directly used as the first rotation matrix;

[0031] After transposing the first text image matrix, the vertical bisector of the transposed matrix is used as the axis of symmetry to perform axis symmetry processing on the matrix elements to obtain the second rotation matrix;

[0032] After performing axis symmetry processing on the matrix elements of the first text image matrix using the vertical bisector as the axis of symmetry, the horizontal bisector of the processed matrix is used as the axis of symmetry to perform axis symmetry processing on the matrix elements to obtain the third rotation matrix;

[0033] After transposing the first text image matrix, the horizontal bisector of the transposed matrix is used as the axis of symmetry to perform axis symmetry processing on the matrix elements to obtain the fourth rotation matrix.

[0034] The second aspect of the present application proposes a photographing and searching method, including:

[0035] A first text image containing a title of a subject to be searched is collected;

[0036] The first text image is subjected to direction correction using any of the above-mentioned text image direction correction methods before subject title recognition is performed;

[0037] The first text image is subjected to recognition.

[0038] The third aspect of the present application proposes a text image direction correction device, including:

[0039] The acquisition module is configured to acquire a first text image;

[0040] The rotation module is configured to rotate the first text image according to N predetermined rotation angles to obtain N rotated images, wherein N is a natural number greater than or equal to 2.

[0041] The estimation module is configured to estimate a correction direction of the first text image according to the N rotated images, wherein the correction direction is a direction after the rotation operation of the first text image for facilitating subsequent image recognition.

[0042] The correction module is configured to correct the direction of the first text image according to the correction direction of the first text image.

[0043] The fourth aspect of the present application provides an electronic device including a processor and a memory, wherein the memory is configured to store a computer executable program, and when the computer executable program is executed by the processor, the processor executes the method.

[0044] The fifth aspect of the present application further provides a computer readable medium storing a computer executable program, and when the computer executable program is executed, the method is implemented.

[0045] (III) Beneficial Effects

[0046] The present application rotates the first text image according to N predetermined rotation angles to obtain N rotated images, estimates the correction direction of the first text image according to the N rotated images, for example, by inputting the N rotated images as feature data into a trained direction estimation model to obtain the correction direction, and corrects the direction of the first text image according to the correction direction of the first text image, thereby ensuring that the direction of the text image is consistent with the recognition direction, improving the accuracy of question recognition, and improving user experience. Compared with the prior art, the present application comprehensively judges the correction direction of the first text image according to the rotated images of multiple directions obtained by rotating the first text image, has higher accuracy, and only needs to input the text image, without the need to detect and recognize the text line, so that the correction direction of the text image can be quickly recognized, the direction of the text image not containing the text can also be corrected, the operation is simple and fast, and the application range is wide.

[0047] The estimation model of the present application can include N neural networks and a judgment module connected with the N neural networks respectively, the N neural networks respectively calculate the probability of the direction of the input rotated image being correct, and the judgment module judges the correction direction of the first text image according to the probability level of the direction of each rotated image being correct. Compared with the traditional convolutional neural network, the recognition effect of the model is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flowchart of a text image direction correction method according to the present application;

[0049] Figures 2a to 2e is a schematic diagram of a rotation operation on a first text image according to the present application;

[0050] Figures 3a to 3d is a schematic diagram of a horizontal bisector of a matrix and a vertical bisector of the matrix according to the present application;

[0051] Figure 4 is a schematic diagram of a direction estimation model according to the present application;

[0052] Figure 5 is a schematic diagram of a neural network according to the present application;

[0053] Figure 6 is a flowchart of a direction correction of a first text image according to the present application;

[0054] Figure 7 is a structural schematic diagram of a text image direction correction device according to the present application;

[0055] Figure 8 is a structural schematic diagram of an electronic device according to an embodiment of the present application;

[0056] Figure 9 is a schematic diagram of a computer-readable recording medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] In the introduction of the specific embodiments, the structural, performance, effect or other characteristics are described in detail in order to make the embodiments fully understood by those skilled in the art. However, it does not exclude that those skilled in the art can implement the present application without the above-mentioned structures, performances, effects or other characteristics in specific cases.

[0058] The flowchart in the accompanying drawings is only an example of flow demonstration, does not represent that all the contents, operations and steps in the flowchart must be included in the scheme of the present application, nor does it represent that the execution must be performed according to the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the inventive concept of the present application.

[0059] The block in the accompanying drawings Figure 1Generally, the functional entities are represented and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different network and / or processing unit devices and / or microcontroller devices.

[0060] The same reference numbers in different drawings represent the same or similar elements, components, or parts and therefore repeated description of the same or similar elements, components, or parts can be omitted below. It should also be understood that, although a first, second, third, etc. denoting adjective can be used herein to describe various devices, elements, components, or parts, these devices, elements, components, or parts should not be limited by these adjectives. That is, a first device can also be referred to as a second device without departing from the essential technical solutions of the present application. In addition, the terms "and / or" and "and / or" mean all combinations of one or more of the listed items.

[0061] The terms involved herein are explained as follows:

[0062] Text image: an image containing text content or pattern content or both, such as a photo of a test paper page, a book page, a book cover, etc.

[0063] Text image direction correction: correcting an image with a direction different from the recognition direction to an image with the same direction as the recognition direction.

[0064] The recognition direction refers to the arrangement direction of the font or pattern set in the recognition process, such as the question recognition. Generally, when the image is normally captured (strictly captured according to the requirements, without deviation or distortion), the arrangement of the question font and image in the image conforms to the reading habit of people, for example, the font is in the normal direction without inversion or skew.

[0065] To solve the above technical problems, the present application provides a text image direction correction method, which rotates a first text image according to N predetermined rotation angles to obtain N rotated images; then, for example, by inputting the N rotated images as feature data into a trained direction estimation model, the correction direction of the first text image is estimated; finally, the direction of the first text image is corrected according to the correction direction of the first text image, so as to ensure that the direction of the text image is consistent with the recognition direction, improve the question recognition accuracy, and improve the user experience.

[0066] The application can adopt an estimation model to estimate the correction direction of the first text image according to the N rotated images, the estimation model comprising N neural networks and a judgment module connected with the N neural networks respectively; the N neural networks respectively calculate the probability of the direction of the input rotated image being correct; and the judgment module judges the correction direction of the first text image according to the probability level of the direction of each rotated image being correct. The direction of the rotated image being correct refers to the direction of the rotated image (obtained after the first text image is rotated by a preset angle) being consistent or close to the required image direction when subsequent recognition is performed. Compared with a traditional convolutional neural network, the recognition effect of the model is more accurate.

[0067] The neural network can comprise a shallow convolutional network and a self-attention neural network connected in sequence; and the self-attention neural network is connected with the judgment module. The shallow convolutional network is used to extract feature data of the rotated image; the self-attention neural network comprises a self-attention module and a binary classification module, and is used to obtain the probability of the direction of the rotated image being correct according to the feature data. Optionally, the shallow convolutional network comprises a plurality of convolutional blocks and a fully connected layer, and each convolutional block can comprise a convolutional layer, a pooling layer, a batch normalization (BN) layer and an excitation ReLU layer.

[0068] To make the objects, technical solutions and advantages of the application clearer, the following further describes the application in detail with reference to the specific embodiments and the accompanying drawings.

[0069] Figure 1 is a flowchart of a text image direction correction method according to the application, as shown in Figure 1 The method comprises the following steps:

[0070] S1, obtaining a first text image;

[0071] This step obtains the first text image to be direction-corrected so as to better perform image recognition and other operations. The first text image can be, for example, a photo containing a title to be searched, which is collected by a user in a photo search scene.

[0072] In this context, the first text image can be an original text image directly obtained from a collection end without any processing, or an image obtained after the original text image is processed (such as filter processing, brightness processing, etc.) to facilitate image recognition.

[0073] The first text image containing the title of the user's search target can be obtained directly by an image collector such as a camera or a scanner, or can be obtained from a memory. The title of the search target in the first text image can only contain text (such as a reading question for high school students), can only contain patterns (such as a question for a kindergarten), or can contain both text and patterns (such as a geometry question), and the present application does not make specific limitations.

[0074] S2, rotating the first text image according to N predetermined rotation angles to obtain N rotated images;

[0075] In order to improve the accuracy of direction estimation, the first text image is rotated according to a plurality of predetermined rotation angles to obtain a plurality of rotated images, and the plurality of rotated images are input into a direction estimation model to estimate the predetermined rotation angle (i.e. the correction direction) to which the first text image needs to be rotated for direction correction.

[0076] The rotation of the first text image can be performed in a synchronous rotation manner, or the rotation effect can be achieved through image processing, for example.

[0077] For example, the step can include:

[0078] S21, converting the first text image into a first text image matrix;

[0079] In a specific example, the first text image can be converted into a corresponding first text image matrix by using python, matlab, etc.

[0080] S22, performing matrix operations on the first text image matrix to obtain N rotation matrices corresponding to the N rotated images.

[0081] The predetermined rotation angle herein refers to an angle corresponding to the rotation of the first text image from a starting direction to a terminal direction. The number N of predetermined rotation angles is a natural number greater than or equal to two. Obviously, the more the number of predetermined rotation angles in the present application, the higher the accuracy of the correction direction estimation of the direction estimation model, but the larger the calculation amount of the model. In order to balance the estimation accuracy and the calculation amount, the number N of predetermined rotation angles is preferably four, including: no rotation of the first text image direction, rotation of the first text image direction by 90 degrees along a first direction, rotation of the first text image direction by 180 degrees along the first direction, and rotation of the first text image direction by 270 degrees along the first direction. The first direction can be a clockwise direction or a counterclockwise direction.

[0082] The following will be described in conjunction with the accompanying drawings Figures 2a to 2eThe rotation process of the first text image is described by taking the first direction as the clockwise direction as an example.

[0083] As Figure 2a The starting direction of the first text image is set in advance for the convenience of describing the rotation process. For example, the starting direction of the first text image is set as the reading direction from left to right of the characters and / or patterns in the first text image, that is, Figure 2a the direction from left to right indicated by arrow OA. Thus, the rotation of the first text image is performed with reference to the starting direction of the first text image. The first preset rotation angle is zero degree. For the first text image without rotation, the first text image matrix is directly taken as the first rotated image matrix, Figure 2b which is the first rotated image obtained by not rotating the starting direction OA of the first text image. The second preset rotation angle is 90 degrees in the clockwise direction relative to OA. For the first text image rotated by 90 degrees in the clockwise direction, the second rotated image matrix is obtained by transposing the first text image matrix and then performing axis symmetry processing on the matrix elements with the vertical bisector of the transposed matrix as the axis of symmetry. Figure 2c which is the second rotated image obtained by rotating the starting direction OA of the first text image by 90 degrees in the clockwise direction.

[0084] The third preset rotation angle is 180 degrees in the clockwise direction relative to OA. For the first text image rotated by 180 degrees in the first direction, the third rotated image matrix is obtained by performing axis symmetry processing on the matrix elements with the vertical bisector of the first text image matrix as the axis of symmetry, and then performing axis symmetry processing on the matrix elements with the horizontal bisector of the processed matrix as the axis of symmetry. Figure 2d which is the third rotated image obtained by rotating the starting direction OA of the first text image by 180 degrees in the clockwise direction. The fourth preset rotation angle is 270 degrees in the clockwise direction relative to OA. For the first text image rotated by 270 degrees in the first direction, the fourth rotated image matrix is obtained by transposing the first text image matrix and then performing axis symmetry processing on the matrix elements with the horizontal bisector of the transposed matrix as the axis of symmetry. Figure 2e which is the fourth rotated image obtained by rotating the starting direction OA of the first text image by 90 degrees in the clockwise direction.

[0085] The horizontal bisector of the matrix refers to the straight line on which the middle row of the matrix is located. For an odd-row matrix, the straight line on which the middle row is located refers to the straight line connecting the elements in the middle row, such as Figure 3a the straight line M1N1 in the middle. For an even-row matrix, the straight line on which the middle row is located refers to the row bisector of the middle two rows of the matrix, such as Figure 3bThe middle straight line M2N2. The vertical bisector of the matrix refers to the straight line where the middle column of the matrix is located. For an odd-numbered column matrix, the straight line where the middle column is located refers to the straight line connecting each element of the middle column of the matrix, such as Figure 3c The middle straight line M3N3. For an even-numbered column matrix, the straight line where the middle column is located refers to the straight line that divides the middle two columns of the matrix, such as Figure 3d The middle straight line M4N4.

[0086] In this paper, the above-mentioned matrix operation can be completed by calling the function cv2.flip in opencv through the interface of python.

[0087] S3, estimating the correction direction of the first text image according to the N rotated images;

[0088] The correction direction refers to the direction of the first text image after the rotation operation for subsequent image recognition. This direction is generally the direction that conforms to the reading habit of the user, that is, the font in the image is positive, not negative or skewed.

[0089] This step can estimate the correction direction of the first text image according to the N rotated images, and the specific implementation manner is not limited. For example, test text can be added to the first text image, and then the test text of the N rotated images can be recognized respectively, and according to the recognition accuracy level, it can be determined which direction of the rotated image is closer to the correct one, that is, the direction of the image corresponding to the most effective recognition scheme is closer.

[0090] This step can also evaluate which direction of the above-mentioned N rotated images can be used as the correction direction of the first text image, that is, determine which one of the preset rotation angles is used for rotating the first image direction to correct. In one specific embodiment, the probability of the direction of each rotated image being correct can be calculated, and the rotation angle corresponding to the rotated image with the maximum probability can be selected to determine the correction direction of the first text image. This embodiment can be realized through the following direction estimation model.

[0091] Exemplarily, the N rotated images can be input into the trained direction estimation model to evaluate the probability of each rotated image being in the correct direction; and the predetermined rotation angle corresponding to the rotated image with the highest probability is taken as the correction direction of the first text image. The probability of the rotated image being in the correct direction refers to the probability of the direction of the rotated image being the recognition direction. The direction estimation model can respectively estimate the probability of each rotated image being in the recognition direction according to each rotated image, and take the predetermined rotation angle corresponding to the direction of the rotated image with the highest probability as the correction direction of the first text image, that is, the target direction to which the direction of the first text image needs to be corrected, so that the problem of being unable to recognize or having a low recognition accuracy due to the inversion or skew of the text and / or pattern in the question recognition process can be avoided or reduced. The recognition direction refers to the direction of the font or pattern of the input image set by the recognition scheme in the question recognition process.

[0092] Optionally, the direction estimation model is generated after the image classification model is trained. The direction of the original image is inferred by identifying the categories of the rotated images of the original image, and the correction direction of the original image is obtained.

[0093] Exemplarily, as shown in Figure 4 The direction estimation model 30 can include N neural networks 31 and a judgment module 32 connected with the N neural networks 31 respectively.

[0094] The N neural networks 31 respectively calculate the probability of the input rotated image being in the correct direction; the neural network 31 includes a shallow convolutional network 311 and a self-attention neural network 312 connected in sequence; and the self-attention neural network 312 is connected with the judgment module 32.

[0095] The judgment module 32 judges the correction direction of the first text image according to the probability level of each rotated image being in the correct direction.

[0096] Exemplarily, in order to improve the recognition accuracy of the direction judgment, the judgment module 32 can comprehensively judge the correction direction of the first text image according to the probability of each rotated image being in the correct direction. For example, the direction of the rotated image with the highest probability is directly determined as the correction direction of the first text image, or the specified direction between the direction of the rotated image with the highest probability and the direction of the rotated image with the second highest probability is taken as the first correction direction of the first text image. Exemplarily, the shallow convolutional network 310 includes a plurality of convolutional blocks (con1, con2, con3) and a fully connected layer, and the shallow convolutional network is used to extract feature data of the rotated image; the self-attention neural network includes a self-attention module and a binary classification module, and the self-attention neural network is used to obtain the probability of the rotated image being in the correct direction according to the feature data; optionally, each convolutional block includes a convolutional layer, a pooling layer, a batch normalization BN layer and an excitation relu layer.

[0097] In one implementation, such as Figure 5 As shown, the shallow convolutional network 310 includes three convolutional blocks: conv1, conv2, and conv3, and a fully connected layer. Each convolutional block consists of a convolutional layer, a pooling layer, a BN (Batch Normalization) layer, and a ReLU (Rectified Linear Units) layer. The input rotated image passes through the three convolutional blocks and the fully connected layer sequentially. For example, the data format of the rotated image matrix is ​​[h1, w1, c1], the data format after passing through the convolutional blocks is [h2, w2, c2], and the data format after passing through the fully connected layer is [T, c3]. Here, h1 and w1 are the sizes of the rotated image resized to a fixed size, such as 512*512, and c1 = 3, indicating that the input image is a color 3-channel image; h2, w2, and c2 are the sizes of the corresponding feature maps of the rotated image, such as 16*16*128; and T and c3 are the sizes of the output image, such as 16*256.

[0098] The self-attention neural network is used to identify the probability that the rotation direction of the image is correct (i.e., consistent with the identified direction) based on feature data. The self-attention neural network 311 may be based on a Transformer model.

[0099] The self-attention neural network 311 includes a self-attention module and a binary classification module. The self-attention module is used to transform the input image into a format that meets the requirements of the binary classification module, and the binary classification module is used to obtain the probability that the orientation of the rotated image is correct.

[0100] Exemplarily, the self-attention module includes a position-related feedforward network and a multi-head self-attention layer. The position-related feedforward network encodes the position information of the input data while encoding the input data in the encoding stage. The multi-head self-attention layer uses a self-attention mechanism to associate the input data and its position in the same input sequence in each sub-layer. In addition, the attention is called multi-head because several attention layers are stacked in parallel with different linear transformations of the same input. This helps the model capture various aspects of the input and improve its expressive power. Exemplarily, the self-attention module has an input data size of 16*256 and an output size of 16*200. The binary classification module is composed of a fully connected layer and an activation layer (such as softmax). This structure realizes parallel processing, shorter training time and higher conversion accuracy without any repeated components. Further, when the maximum value of the direction correct probability of each rotated image is less than the preset probability, the corrected direction of the first text image is not determined, but the N predetermined rotation angles of the next rotation are determined based on the direction correct probability of each rotated image, and the above probability calculation and judgment are continued until the maximum value of the direction correct probability is less than the preset probability, and then the corrected direction is determined.

[0101] Exemplarily, if the predetermined rotation angle of the rotated image A is counterclockwise rotation by 45 degrees; if the predetermined rotation angle of the rotated image B is counterclockwise rotation by 90 degrees; and if the maximum value of the direction correct probability of the rotated image A is calculated subsequently, and the direction correct probability of the rotated image B is second (or both are about the same), it can be judged that the corrected direction of the first rotated image is between the directions of the rotated image A and the rotated image B, and is close to the direction of the rotated image A, i.e., the corrected direction is between counterclockwise rotation by 45 degrees and counterclockwise rotation by 90 degrees, and is closer to counterclockwise rotation by 90 degrees. If the direction correct probability of the rotated image A is less than the preset probability (the recognition accuracy is still unacceptable when the first text image is corrected by counterclockwise rotation by 45 degrees corresponding to the rotated image A), the angle of the second rotation is determined within this range (or the range can be appropriately expanded according to the actual situation), for example, counterclockwise rotation by 60 degrees, 75 degrees, 80 degrees, 85 degrees, etc., and the direction correct probability of each rotated image after the second rotation is calculated, and the iteration is continued similarly subsequently until the maximum value of the direction correct probability is greater than or equal to the preset probability, and then the corrected direction is determined according to the rotation angle of the rotated image with the maximum probability.

[0102] Figure 6 The flowchart for estimating the corrected direction of the first text image of the present application is shown in Figure 6 Exemplarily, the method includes:

[0103] S61, acquiring a first text image;

[0104] S62, rotating the first text image according to four predetermined rotation angles to obtain four rotated images.

[0105] The four predetermined rotation angles include: no rotation of the first text image direction, rotation of the first text image direction by 90 degrees along a first direction, rotation of the first text image direction by 180 degrees along the first direction, and rotation of the first text image direction by 270 degrees along the first direction. The first direction can be a clockwise direction or a counterclockwise direction. Figure 6 Each arrow direction represents a predetermined rotation direction.

[0106] S63, input the four rotated images into a shallow convolutional network and a self-attention module respectively, estimate the probability of the direction of each rotated image being correct, and obtain the estimated result of each rotated image;

[0107] The shallow convolutional network is marked as CNN. The estimated result of each rotated image is represented by result 1 to result 4.

[0108] S64, determine the correction direction of the first text image according to the estimated result of each rotated image;

[0109] In order to improve the accuracy of the direction estimation model, the direction estimation model can be pre-trained.

[0110] The training set is obtained by data augmentation of historical text images. The historical text images can be images containing questions, such as test papers, book pages, and covers. The data augmentation refers to adding Gaussian noise and salt and pepper noise to the original image, and then performing crop operation on the image.

[0111] S4, correct the direction of the first text image according to the correction direction of the first text image.

[0112] For example, according to the preset angles of 0 degrees, 90 degrees, 180 degrees and 270 degrees, if the correction direction is to not rotate the first text image, the first text image is not corrected in this step and is directly identified; if the correction direction is to rotate the first text image by 90 degrees along the first direction, the first text image matrix is transposed in this step, and then the horizontal bisector of the transposed matrix is used as the axis of symmetry to perform axis symmetry processing on the matrix elements, to obtain the corrected first text image, and then identification is performed; if the correction direction is to rotate the first text image by 180 degrees along the first direction, the vertical bisector of the first text image matrix is used as the axis of symmetry to perform axis symmetry processing on the matrix elements in this step, and then the horizontal bisector of the processed matrix is used as the axis of symmetry to perform axis symmetry processing on the matrix elements, to obtain the corrected first text image, and then identification is performed; if the correction direction is to rotate the first text image by 270 degrees along the first direction, the first text image matrix is transposed in this step, and then the vertical bisector of the transposed matrix is used as the axis of symmetry to perform axis symmetry processing on the matrix elements, to obtain the corrected first text image, and then identification is performed. The first direction is the same as the first direction in step S2, that is, the first direction of the rotation operation is the clockwise direction, and the first direction in this step is also the clockwise direction, and vice versa.

[0113] The present application rotates the first text image according to N predetermined rotation angles synchronously, obtains N rotated images, inputs the N rotated images as feature data into the trained direction estimation model to identify the correction direction of the first text image, and corrects the direction of the first text image according to the correction direction of the first text image, so as to ensure that the direction of the text image is consistent with the identification direction, improve the question identification accuracy, and improve the user experience. Compared with the prior art, the present application comprehensively identifies the correction direction of the first text image according to the multiple direction rotated images obtained by synchronous rotation of the first text image, has higher accuracy, and only needs to input the text image, without detecting and identifying the text line, so that the correction direction of the text image can be quickly identified, and the correction direction of the text image not containing text can also be identified, which is simple to operate, fast to identify, and has a wide range of applications.

[0114] The present application also provides a photographing question searching method, which comprises the following steps:

[0115] S101, collecting a first text image containing a question title to be searched;

[0116] S102, correcting the direction of the first text image by using the text image direction correction method according to any one of the above embodiments before identifying the question title;

[0117] S103, identifying the first text image.

[0118] Figure 7 is a structural framework diagram of a text image direction correction device provided by the present application, as shown in Figure 7 the device comprises:

[0119] an acquisition module 71, configured to acquire a first text image;

[0120] a rotation module 72, configured to rotate the first text image according to N predetermined rotation angles to obtain N rotated images, wherein N is a natural number greater than or equal to two;

[0121] an estimation module 73, configured to estimate a correction direction of the first text image according to the N rotated images, wherein the correction direction refers to a direction after the first text image is rotated for subsequent image recognition;

[0122] a correction module 74, configured to correct the direction of the first text image according to the correction direction of the first text image.

[0123] Illustratively, the estimation module 73 is configured to input the N rotated images into a trained direction estimation model to evaluate the direction correctness probability of each rotated image, and take the direction of the rotated image with the highest probability as the correction direction of the first text image.

[0124] Optionally, the direction estimation model is generated after training an image classification model.

[0125] Illustratively, the direction estimation model comprises N neural networks and a judgment module connected with the N neural networks respectively; the N neural networks respectively calculate the direction correctness probability of the input rotated image; and the judgment module judges the correction direction of the first text image according to the direction correctness probability level of each rotated image.

[0126] Optionally, the neural network comprises a shallow convolutional network and a self-attention neural network connected in sequence, and the self-attention neural network is connected with the judgment module; the shallow convolutional network is configured to extract feature data of the rotated image; and the self-attention neural network is configured to obtain the direction correctness probability of the rotated image according to the feature data.

[0127] Optionally, the shallow convolutional network comprises a plurality of convolutional blocks and a fully connected layer.

[0128] Optionally, each convolutional block comprises a convolutional layer, a pooling layer, a batch normalization layer and an excitation layer.

[0129] Illustratively, the self-attention neural network is based on a Transformer model.

[0130] Optionally, the self-attention neural network includes: a self-attention module and a binary classification module, wherein the self-attention module is used to transform the input image into a specification that meets the requirements of the binary classification module, and the binary classification module is used to obtain the probability that the orientation of the rotated image is correct;

[0131] Optionally, the binary classification module includes a fully connected layer and a softmax layer.

[0132] The rotation module 72 includes:

[0133] A conversion module is used to convert the first text image into a first text image matrix;

[0134] The operation module is used to perform matrix operations on the first text image matrix to obtain N rotation matrices corresponding to the N rotated images.

[0135] For example, the predetermined rotation angle is four, including: the first text image is not rotated, the first text image is rotated 90 degrees along the first direction, the first text image is rotated 180 degrees along the first direction, and the first text image is rotated 270 degrees along the first direction.

[0136] The operation module is used to directly use the first text image matrix as the first rotated image matrix; after transposing the first text image matrix, the vertical bisector of the transposed matrix is ​​used as the axis of symmetry to perform axial symmetry processing on the matrix elements to obtain the second rotated image matrix; after performing axial symmetry processing on the matrix elements using the vertical bisector of the first text image matrix as the axis of symmetry, the horizontal bisector of the processed matrix is ​​used as the axis of symmetry to perform axial symmetry processing on the matrix elements to obtain the third rotated image matrix; after transposing the first text image matrix, the horizontal bisector of the transposed matrix is ​​used as the axis of symmetry to perform axial symmetry processing on the matrix elements to obtain the fourth rotated image matrix.

[0137] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. The electronic device includes a processor and a memory. The memory is used to store a computer-executable program. When the computer program is executed by the processor, the processor executes a text image orientation correction method.

[0138] like Figure 8 As shown, the electronic device is embodied in the form of a general-purpose computing device. There can be one or more processors working collaboratively. This invention also does not preclude distributed processing, meaning that processors can be distributed across different physical devices. The electronic device of this invention is not limited to a single entity, but can also be the sum of multiple physical devices.

[0139] The memory stores a computer-executable program, typically machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps of the method.

[0140] The memory includes volatile memory, such as random access memory (RAM) and / or cache memory, and may also be non-volatile memory, such as read-only memory (ROM).

[0141] Optionally, in this embodiment, the electronic device further includes an I / O interface for exchanging data with external devices. The I / O interface can represent one or more of several bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0142] It should be understood that Figure 8 The electronic device shown is merely one example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as displays, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. Any electronic device capable of executing a computer-readable program in memory to implement the method of the present invention or at least some steps of the method can be considered as an electronic device covered by the present invention.

[0143] Figure 9 This is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention. Figure 9 As shown, a computer-readable recording medium stores a computer-executable program, which, when executed, implements the text image orientation correction method described above. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0144] The program code may be implemented in any of various high level languages, including an object oriented language such as Java, C++, or the like, and / or a conventional procedural language, such as the "C" language or similar. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0145] From the above description of the embodiments, it is easy for those skilled in the art to understand that the present application can be implemented by hardware capable of executing a specific computer program, such as a system of the present application, and electronic processing units, servers, clients, mobile phones, control units, processors, etc. included in the system, and can also be implemented by a vehicle including at least part of the above-mentioned system or components. The present application can also be implemented by computer software for executing the method of the present application, such as control software executed by microprocessors, electronic control units, clients, server ends, etc. of the live device. It should be noted that the computer software for executing the method of the present application is not limited to being executed by one or specific hardware entity, but can also be implemented in a distributed manner by unspecified hardware. As for the computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or can be stored in a distributed manner on a network, as long as it can enable electronic devices to execute the method according to the present application.

[0146] The above-described specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application, and it should be understood that the present application is not inherently related to any specific computer, virtual device or electronic device, and various general-purpose devices can also implement the present application. The above-described is only a specific embodiment of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for text image orientation correction, characterized in that, The method comprises the following steps: rotating the obtained first text image according to N predetermined rotation angles to obtain N rotated images, wherein N is a natural number greater than or equal to 2, and the method comprises the following steps: converting the first text image into a first text image matrix, and performing matrix operation on the first text image matrix to obtain N rotation matrices corresponding to the N rotated images, wherein in the matrix operation, the horizontal bisector of the transposed matrix refers to a straight line on which the middle row of the matrix is located, and the vertical bisector of the transposed matrix refers to a straight line on which the middle column of the matrix is located; estimating the correction direction of the first text image according to the N rotated images, which comprises the following steps: inputting the N rotated images into a trained direction estimation model to estimate the probability of the direction of each rotated image being correct, and taking the direction of the rotated image with the highest probability as the correction direction of the first text image; wherein the correction direction refers to the target direction required for the first text image after the rotation operation for the subsequent image recognition, the direction estimation model is generated after training the image classification model, and the direction estimation model comprises N neural networks and judgment modules connected with the N neural networks respectively, the neural network comprises a shallow convolutional network and a self-attention neural network connected in sequence, the self-attention neural network is connected with the judgment module and the self-attention module comprises a position-related feedforward network and a multi-head self-attention layer; the N neural networks calculate the probability of the direction of the input rotated image being correct respectively, the judgment module estimates the correction direction of the first text image according to the probability level of the direction of each rotated image being correct, and the direction of the rotated image being correct refers to that the direction of the rotated image is consistent or close to the required image direction in the subsequent image recognition; performing direction correction on the first text image according to the correction direction of the first text image.

2. The method of claim 1, wherein, The neural network further comprises: the shallow convolutional network is used for extracting feature data of the rotated image; the self-attention neural network is used for obtaining the probability of the direction of the rotated image being correct according to the feature data; the shallow convolutional network comprises a plurality of convolutional blocks and a fully connected layer, and each convolutional block comprises a convolutional layer, a pooling layer, a batch normalization layer and an excitation layer.

3. The method of claim 2, wherein, Further comprising: the self-attention neural network is based on a Transformer model; the self-attention neural network comprises a self-attention module and a binary classification module, the self-attention module is used for converting the input image into a specification meeting the requirements of the binary classification module, the binary classification module is used for obtaining the probability of the direction of the rotated image being correct, and the binary classification module comprises a fully connected layer and a softmax layer.

4. The method of claim 1, wherein, The predetermined rotation angles comprise four predetermined rotation angles, i.e., no rotation of the first text image, rotation of the first text image by 90 degrees along a first direction, rotation of the first text image by 180 degrees along the first direction, and rotation of the first text image by 270 degrees along the first direction.

5. The method of claim 4, wherein the matrix operation on the first text image matrix to obtain N rotation matrices corresponding to the N rotated images comprises: the first text image matrix is directly taken as a first rotated image matrix; The vertical bisector of the first text image matrix is taken as the axis of symmetry to perform axis symmetry processing on the matrix elements, and then the horizontal bisector of the processed matrix is taken as the axis of symmetry to perform axis symmetry processing on the matrix elements, to obtain a third rotated image matrix. The vertical bisector of the first text image matrix is taken as the axis of symmetry to perform axis symmetry processing on the matrix elements, and then the horizontal bisector of the processed matrix is taken as the axis of symmetry to perform axis symmetry processing on the matrix elements, to obtain a third rotated image matrix. The vertical bisector of the first text image matrix is taken as the axis of symmetry to perform axis symmetry processing on the matrix elements, and then the horizontal bisector of the processed matrix is taken as the axis of symmetry to perform axis symmetry processing on the matrix elements, to obtain a third rotated image matrix.

6. A method for taking a picture to search for a question, characterized in that, It comprises: Collecting a first text image containing a title to be searched; Before performing title recognition, the first text image is directionally corrected using the method of any one of claims 1-5; The first text image is recognized.

7. A text image direction correction apparatus characterized by comprising: It comprises: A rotation module for rotating the obtained first text image according to N predetermined rotation angles to obtain N rotated images, wherein N is a natural number greater than or equal to two, including: converting the first text image into a first text image matrix, and performing matrix operations on the first text image matrix to obtain N rotation matrices corresponding to the N rotated images, wherein the horizontal bisector of the transposed matrix refers to a straight line where the middle row of the matrix is located, and the vertical bisector of the transposed matrix refers to a straight line where the middle column of the matrix is located; An estimation module for estimating the correction direction of the first text image according to the N rotated images, including: inputting the N rotated images into a trained direction estimation model to evaluate the probability of the direction of each rotated image being correct, and taking the direction of the rotated image with the highest probability as the correction direction of the first text image; wherein the correction direction refers to the target direction required for the first text image after the subsequent image recognition rotation operation, the direction estimation model is generated after training the image classification model, and the direction estimation model comprises N neural networks and a judgment module connected with the N neural networks, the neural network comprises a shallow convolutional network and a self-attention neural network connected in turn, the self-attention neural network is connected with the judgment module and the self-attention module comprises a position-related feedforward network and a multi-head self-attention layer; the N neural networks calculate the probability of the direction of the input rotated image being correct respectively, the judgment module judges the correction direction of the first text image according to the probability level of the direction of each rotated image being correct, and the direction of the rotated image being correct refers to the direction of the rotated image being consistent or close to the required image direction in the subsequent image recognition; A correction module for directionally correcting the first text image according to the correction direction of the first text image.

8. An electronic device comprising a processor and a memory, the memory being configured to store a computer executable program, characterized in that: When the computer executable program is executed by the processor, the processor executes the method of any one of claims 1-5.

9. A computer readable medium storing a computer executable program, characterized in that, When the computer executable program is executed, the method of any one of claims 1-5 is implemented.

Images