Image processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method, device, electronic device and storage medium. A target image is obtained, the target image includes multiple lines of text, each line of text in the multiple lines of text includes at least one character; the characters in the target image are recognized by a pre-trained text recognition model, the text recognition model includes a feature mapping layer, a feature extraction layer, an adaptive clustering layer and an output layer, wherein the feature mapping layer performs feature mapping on the target image to obtain mapping information corresponding to the target image, the feature extraction layer performs feature extraction on the target image to obtain feature information corresponding to the target image, the adaptive clustering layer obtains a feature matrix based on the mapping information and feature information, and the output layer obtains a probability matrix corresponding to the target image based on the feature matrix; according to the probability matrix corresponding to the target image, the recognition result of the characters in the target image is obtained, and the text in the image can be accurately recognized, with fast recognition speed and high accuracy.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Currently, image-based text recognition is widely used. Usually, before performing text recognition, it is necessary to detect the number of text lines included in the image and then perform recognition based on each line of text.

[0003] However, the method of first detecting and then recognizing may have problems with missed recognition or over-recognition, and for images with complex text presentation or background, such as straight text, tilted text, and curved text on the image, the method of first detecting and then recognizing may introduce more errors, slow recognition speed, and low accuracy. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides an image processing method, which can accurately recognize text in an image with fast recognition speed and high accuracy.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

[0006] Acquire a target image, the target image includes multiple lines of text, each line of the multiple lines of text includes at least one character;

[0007] Recognize characters in a target image through a pre-trained text recognition model, the text recognition model includes a feature mapping layer, a feature extraction layer, an adaptive clustering layer and an output layer, wherein the feature mapping layer performs feature mapping on the target image to obtain mapping information corresponding to the target image, the feature extraction layer performs feature extraction on the target image to obtain feature information corresponding to the target image, the adaptive clustering layer obtains a feature matrix based on the mapping information and the feature information, and the output layer obtains a probability matrix corresponding to the target image based on the feature matrix;

[0008] According to the probability matrix corresponding to the target image, the recognition result of the characters in the target image is obtained.

[0009] In a second aspect, an embodiment of the present disclosure provides an image processing device, including:

[0010] An acquisition unit, used for acquiring a target image, the target image includes multiple lines of text, each line of the multiple lines of text includes at least one character;

[0011] A first recognition unit is used to recognize characters in a target image through a pre-trained text recognition model, wherein the text recognition model includes a feature mapping layer, a feature extraction layer, an adaptive clustering layer and an output layer, wherein the feature mapping layer performs feature mapping on the target image to obtain mapping information corresponding to the target image, the feature extraction layer performs feature extraction on the target image to obtain feature information corresponding to the target image, the adaptive clustering layer obtains a feature matrix based on the mapping information and the feature information, and the output layer obtains a probability matrix corresponding to the target image based on the feature matrix;

[0012] The second recognition unit is used to obtain the recognition result of the characters in the target image according to the probability matrix corresponding to the target image.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0014] Processor; and

[0015] Memory for storing programs,

[0016] The program includes instructions, and when the instructions are executed by a processor, the processor executes the above-mentioned image processing method.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the above-mentioned image processing method.

[0018] In a fifth aspect, a computer program product includes a computer program, wherein the computer program implements the above-mentioned image processing method when executed by a processor.

[0019] The present disclosure provides an image processing method, device, electronic device and storage medium. The image processing method specifically includes: obtaining a target image, the target image includes multiple lines of text, each line of text in the multiple lines of text includes at least one character; recognizing the characters in the target image through a pre-trained text recognition model, the text recognition model includes a feature mapping layer, a feature extraction layer, an adaptive clustering layer and an output layer, wherein the feature mapping layer performs feature mapping on the target image to obtain mapping information corresponding to the target image, the feature extraction layer performs feature extraction on the target image to obtain feature information corresponding to the target image, the adaptive clustering layer obtains a feature matrix based on the mapping information and feature information, and the output layer obtains a probability matrix corresponding to the target image based on the feature matrix; according to the probability matrix corresponding to the target image, the recognition result of the characters in the target image is obtained, and the text in the image can be accurately recognized with fast recognition speed and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0023] Figure 2 A flowchart of a text recognition model training method provided in an embodiment of the present disclosure;

[0024] Figure 3 A structural diagram of a text recognition model provided by an embodiment of the present disclosure;

[0025] Figure 4 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0026] Figure 5 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0031] At present, text recognition in natural scenes is a very challenging subject. Text recognition is the process of identifying character sequences in pictures including text, that is, the process of identifying all characters included in the image. In natural scenes, in addition to factors such as complex background and illumination changes, the complexity of the recognition output space is also a major difficulty. Since text is composed of letters or characters of an unfixed number, natural scene text recognition needs to recognize sequences of unfixed length from pictures. At present, there are usually two recognition methods. One is based on a bottom-up strategy, which divides the recognition problem into character detection, character recognition and character combination, and solves them one by one. The other is based on a strategy of overall analysis, that is, a sequence-to-sequence method, which first encodes the image and then decodes the sequence to directly obtain the entire string. However, the above two methods have the following problems: Although the first method is effective, it requires character-level annotation, that is, the position and information of each character on the input image need to be manually annotated, which requires a lot of manpower. Although the second method is simple to annotate, it only needs to transcribe the string, but its output recognition result may have the situation of multiple recognized characters or missed recognized characters.

[0032] Secondly, general text images can be roughly divided into three types according to their writing direction and trend, namely normal text (straight text), angled slanted text and curved text, etc. Normal text usually refers to writing from left to right, with all the characters roughly on a straight line and the straight line almost coinciding with the horizontal direction. Angle-slanted text refers to writing from left to right, with all the characters roughly on a straight line and the straight line having a certain angle with the horizontal direction, that is, a slanted line of characters. Curved text refers to an arch-type, written from left to right, but the characters are almost not on the same straight line, and the center points of the characters are roughly connected to form a curve. As mentioned above, common text recognition operations are all for single-line text images, so text detection operations are generally required. As a preliminary step, a specific single line of text is obtained. Generally speaking, for more conventional text images, such as normal text, its text line distribution may be relatively standard, and detection can also achieve good results. Therefore, the detection and recognition operations generally do not have too much error. However, for more complex text images, there may be an obvious hierarchical structure. The following is an example of an image including multiple questions. The image is composed of multiple questions to form a page. The questions can be better detected and distinguished by detection boxes, but each question contains multiple lines, and each line has different lengths (some are long and some are short). Some lines are extremely long and wide (the length is much greater than the width), which increases the difficulty of detection. The above detection and recognition methods may have errors and the recognition results are inaccurate.

[0033] In summary, text recognition has a very wide range of applications. The above two methods have different problems. The character-based method has a high annotation cost, and the sequence-based method may have the problem of missing or over-recognizing characters. The text represented on different text images includes straight text, inclined text and curved text. Most recognition methods are for single-line text and require detection as a pre-operation. However, for images with more complex layouts, the detection plus recognition method may introduce more errors and the recognition accuracy is relatively low.

[0034] In response to the above technical problems, the embodiments of the present disclosure provide an image processing method, which is specifically described in detail through one or more of the following embodiments.

[0035] Specifically, the image processing method can be executed by a terminal or a server. Specifically, the terminal or the server can recognize characters in the target image through a text recognition model. The execution subject of the training method of the text recognition model and the execution subject of the character recognition method can be the same or different.

[0036] For example, in an application scenario, such as Figure 1 As shown, the server 12 trains the text recognition model. The terminal 11 obtains the trained text recognition model from the server 12, and the terminal 11 recognizes the characters in the target image through the trained text recognition model. The target image may be obtained by photographing the terminal 11. Alternatively, the target image is an image obtained by the terminal 11 after performing image processing on a preset image, and the preset image may be obtained by photographing the terminal 11, or the preset image may be obtained by the terminal 11 from other devices. Here, no specific limitation is made to other devices.

[0037] In another application scenario, the server 12 trains the text recognition model. Further, the server 12 recognizes characters in the target image through the trained text recognition model. The way in which the server 12 obtains the target image can be similar to the way in which the terminal 11 obtains the target image as described above, which will not be repeated here.

[0038] In another application scenario, the terminal 11 trains the text recognition model. Further, the terminal 11 recognizes characters in the target image using the trained text recognition model.

[0039] It is understandable that the text recognition model training method and image processing method provided by the embodiments of the present disclosure are not limited to the several possible scenarios described above. Since the trained text recognition model can be applied to the image processing method, before introducing the image processing method, the text recognition model training method can be introduced below.

[0040] The following introduces a text recognition model training method, that is, a text recognition model training process, by taking the server 12 training a text recognition model as an example. It is understandable that the text recognition model training method is also applicable to the scenario where the terminal 11 trains a text recognition model.

[0041] Figure 2 A flowchart of a text recognition model training method provided in an embodiment of the present disclosure specifically includes the following steps: Figure 2 The following steps S210 to S230 are shown:

[0042] S210: Acquire a sample image and characters in the sample image.

[0043] Optionally, the above S210 specifically includes: obtaining a sample image, the sample image including multiple lines of text; marking the characters included in each line of the multiple lines of text, and determining the characters included in each line of text in the sample image, and multiple lines of text belonging to the same sample image are separated by preset characters.

[0044] It is understandable that multiple sample images are obtained. The sample images can be understood as text images to be recognized. Each of the multiple sample images includes multiple text lines. The text lines in the sample images can be expressed in the form of straight text, tilted text, and curved text, etc. For example, any image includes multiple lines of curved text. After obtaining multiple sample images, the multiple sample images are scaled to the same size, and then all characters included in each of the multiple sample images are marked. In the process of marking the characters in the sample images, the characters in the sample images can also be determined. The characters in the marked sample images are used as standard characters and as a reference for the character recognition results output during the training process of the text recognition model, thereby updating the network parameters of the text recognition model. It is understandable that if the sample image only includes a single line of text, the characters in the single line of text can be directly marked. If the sample image includes multiple lines of text, the characters marked between the multiple lines of text in the same sample image are separated by preset characters. The preset character can be "\n", which is used to connect the multiple lines of text in the sample image in series. For example, sample image 1 includes 2 lines of text, the characters marked in the first line of text are "AAA", and the characters marked in the second line of text are "BBBBB". The character sequence in sample image 1 is recorded as "AAA\nBBBBB". It is understandable that a dictionary is established based on the character information marked in multiple sample images, and the dictionary includes the characters in multiple sample images. If the multiple sample images include text, then the established dictionary includes multiple independent and non-repeating texts. Multiple independent and non-repeating texts can be obtained by collecting and operating the marked character sequence into single characters. The established dictionary can be used as a character database for subsequent character recognition based on the text recognition model, and the specific characters are determined in the character recognition library according to the recognition probability.

[0045] S220: Input the sample image into the constructed text recognition model to obtain the preset characters corresponding to the sample image.

[0046] It can be understood that, based on the above S210, after obtaining the sample image and the characters in the sample image, the sample image is input into the constructed text recognition model to obtain the preset characters corresponding to the sample image. The preset characters can be understood as the characters in the sample image recognized by the text recognition model, and the preset characters may not be accurate recognition results.

[0047] For example, see Figure 3 , Figure 3This is a structural diagram of a text recognition model provided by an embodiment of the present disclosure. The constructed text recognition model 300 includes a feature mapping layer 310, a feature extraction layer 320, an adaptive clustering layer 330 and an output layer 340. The feature mapping layer 310 includes multiple sub-feature mapping layers, and the multiple sub-feature mapping layers are connected in sequence. The feature extraction layer 320 can specifically be a network layer composed of a feature point extraction algorithm, and the feature extraction algorithm can specifically be a scale-invariant feature transform (SIFT) algorithm. The adaptive clustering layer 330 includes an encoder, a first convolutional layer and a second convolutional layer. The output layer 340 can be a transcription layer based on the attention mechanism (Attention). The data flow inside the text recognition model includes: the feature mapping layer 310 performs feature mapping on the sample image to obtain mapping information corresponding to the sample image, the feature extraction layer 320 performs feature extraction on the sample image to obtain feature information corresponding to the sample image, the adaptive clustering layer 330 obtains a feature matrix based on the mapping information output by the feature mapping layer 310 and the feature information output by the feature extraction layer 320, and finally the output layer 340 calculates the similarity probability between each character in the feature matrix and the characters in the dictionary established above based on the feature matrix output by the adaptive clustering layer 330 to obtain a probability matrix corresponding to the sample image. After the text recognition model outputs the probability matrix, the final recognition result is determined according to the probability matrix. Specifically, the final recognition result can be obtained according to a greedy algorithm or a beam search algorithm. The final recognition result is the preset character, and the character corresponding to the maximum probability in the probability matrix can be determined as the preset character.

[0048] S230: Update the network parameters of the text recognition model according to the preset characters, the characters in the sample image and the preset loss function.

[0049] It can be understood that on the basis of the above S220, after the sample image is input into the text recognition model, the preset characters included in the sample image are obtained, and the text recognition model is updated according to the preset characters, the characters in the sample image and the preset loss function. For example, the character marked in the sample image 1 is "Peng", and the preset character may be "Peng", "bird" or "peng". If the recognition result is "Peng", it means that the result recognized by the text recognition model is accurate, wherein the preset loss function can be a multi-classification cross entropy loss function. The type of loss function is not limited and can be determined by the user according to the actual usage needs.

[0050] A method for training a text recognition model provided by an embodiment of the present disclosure obtains a sample image, marks the characters in the sample image, inputs the sample image into a constructed text recognition model, obtains preset characters corresponding to the sample image, and then updates the network parameters of the text recognition model according to the preset characters, the characters in the sample image and a preset loss function to obtain the text recognition model. The text recognition model trained by this method has high recognition accuracy and a short training cycle of the model.

[0051] Based on the above embodiments, Figure 4 A flowchart of an image processing method provided in an embodiment of the present disclosure is provided, and is described by taking an example of a terminal recognizing characters in a target image according to a text recognition model, and specifically includes the following steps: Figure 4 Steps S410 to S430 shown:

[0052] S410: Acquire a target image, where the target image includes multiple lines of text, and each line of the multiple lines of text includes at least one character.

[0053] It can be understood that after the terminal obtains the text recognition model, it obtains the target image, which includes multiple lines of text. The multiple lines of text may include multiple lines of straight text, oblique text and curved text. Each line of text includes at least one character, that is, only lines with characters are called text lines.

[0054] S420. Recognize characters in a target image through a pre-trained text recognition model, wherein the text recognition model includes a feature mapping layer, a feature extraction layer, an adaptive clustering layer and an output layer, wherein the feature mapping layer performs feature mapping on the target image to obtain mapping information corresponding to the target image, the feature extraction layer performs feature extraction on the target image to obtain feature information corresponding to the target image, the adaptive clustering layer obtains a feature matrix based on the mapping information and the feature information, and the output layer obtains a probability matrix corresponding to the target image based on the feature matrix.

[0055] It can be understood that the obtained target image is input into the text recognition model that has been trained as mentioned above. The text recognition model directly recognizes the characters in the multiple lines of text in the target image. There is no need to detect the multiple lines of text to determine each line of text and use one line of text as a benchmark for character recognition. That is to say, the text recognition model provided by the present disclosure only needs to perform the recognition step, while the conventional method requires two steps, namely detection and recognition. However, the conventional method will bring new errors during detection, such as incomplete detection such as multiple detection and missed detection, which will affect the subsequent recognition results. In addition, there will be contextual relationships between texts that are not in the same line. Only detecting one line of text will affect the accuracy of the recognition results. The text recognition model provided by the present disclosure directly uses the target image including multiple lines of text as input to directly perform character recognition. There is no need to perform the detection step. While effectively reducing the recognition error, it can also make full use of the contextual information of multiple lines of text. It can also reduce the recognition difficulty of curved text to a certain extent and improve the accuracy of the recognition results.The text recognition model includes a feature mapping layer, a feature extraction layer, an adaptive clustering layer and an output layer. The internal process of the text recognition model includes: the feature mapping layer performs feature mapping on the target image to obtain mapping information corresponding to the target image. The mapping information is used to characterize the global feature information of the target image, that is, to extract relevant information of the characters in the target image and remove background information, so as to facilitate subsequent character recognition based on the feature information of the extracted characters. For example, the size of the target image is 4H*4W, where H represents height and W represents width. Both H and W are greater than 1. After input into the feature mapping layer, a set of mapping information of H*W*512 is obtained, where 512 refers to the number of channels; the feature extraction layer performs feature extraction on the target image to obtain feature information corresponding to the target image. The feature extraction layer can be the network layer corresponding to the sift feature extraction algorithm. The feature information obtained is the coordinates of a series of feature points. The feature information can specifically include feature points that are more prominent in a local range. A preset number of prominent feature points can be retained according to user needs. The preset number can be set by yourself. For example, the preset number can be 1024, that is, 1024 feature points are extracted from the sample image; after obtaining the feature information and mapping information, the adaptive clustering layer obtains the feature matrix based on the mapping information and feature information. The feature matrix includes the category, coordinate information and mapping information corresponding to the character. The category corresponding to the character refers to the target text line where the character is located, the coordinate information refers to the coordinate of the character in the target text line, and the mapping information is the mapping information of the character extracted by the above feature mapping layer. The mapping information can also be called a feature mapping vector. The feature matrix can be N*512 in size, where N represents the maximum possible number of characters corresponding to an image, and N can be a set fixed value. N is calculated as the product of the number of rows and the number of characters included in each row. For example, it is set that the input target image has a maximum of 20 rows, and each row includes a maximum of 10 characters. At this time, the obtained N is 20*10; after obtaining the feature matrix, the output layer obtains the probability matrix corresponding to the target image based on the feature matrix. The probability matrix can be understood as a matrix composed of the similarity probabilities between the characters included in the target image and any character in the dictionary. The output layer can specifically be a transcription layer based on the attention mechanism.

[0056] S430: Obtain recognition results of characters in the target image according to the probability matrix corresponding to the target image.

[0057] It is understandable that, based on the above S420, after determining the probability matrix corresponding to the target image, a greedy algorithm or a beam search algorithm is used based on the probability matrix to obtain the final recognition result, that is, to output all the characters included in the target image.

[0058] An image processing method provided by an embodiment of the present disclosure directly uses a target image including multiple lines of text as the input of a text recognition model for character recognition. There is no need to perform a detection step, that is, there is no need to split the multiple lines of text into single lines of text. The multiple lines of text can be directly recognized, and while effectively reducing recognition errors, the contextual information of the multiple lines of text can be fully utilized. To a certain extent, the difficulty of recognizing curved text is also reduced. The method is suitable for complex application scenarios, improves the accuracy of recognition results, can accurately recognize characters in images, has a fast recognition speed, and has a high recognition accuracy.

[0059] Optionally, the feature mapping layer includes multiple sub-feature mapping layers, and the multiple sub-feature mapping layers are connected in sequence.

[0060] For example, see Figure 3 The feature mapping layer 310 includes 4 sub-feature mapping layers, specifically including a first sub-feature mapping layer, a second sub-feature mapping layer, a third sub-feature mapping layer and a fourth sub-feature mapping layer. The feature mapping layer 310 can be a residual network layer. The specific feature mapping layer 310 can be understood as a Resnet18 network layer. The sub-feature mapping layer can be understood as a Block in the Resnet18 network layer. Each block is composed of several convolutional layers. Each block performs a convolution operation. The output of each block is the input of the next block, that is, each block is connected in sequence. The step size of the convolutional layer corresponding to each block is set in the feature mapping layer 310.

[0061] Optionally, feature mapping is performed on the target image through a feature mapping layer to obtain mapping information corresponding to the target image, including: feature mapping the target image through a first sub-feature mapping layer to obtain first mapping information, and the size of the first mapping information is the same as the size of the target image; feature mapping the first mapping information through a second sub-feature mapping layer to obtain second mapping information, and the size of the second mapping information is half of the size of the first mapping information; feature mapping the second mapping information through a third sub-feature mapping layer to obtain third mapping information, and the size of the third mapping information is the same as the size of the second mapping information; feature mapping the third mapping information through a fourth sub-feature mapping layer to obtain fourth mapping information, and the size of the fourth mapping information is half of the size of the third mapping information, and the fourth mapping information is mapping information corresponding to the target image.

[0062] It can be understood that the feature mapping layer performs feature mapping on the target image and obtains the specific process of mapping information corresponding to the target image, including: the first sub-feature mapping layer performs feature mapping on the target image to obtain first mapping information, the size of the first mapping information is the same as the size of the target image, and the mapping information can also be understood as a mapping feature map. For example, the size of the above-mentioned target image is 4H*4W, and the step size of the convolution in the first sub-feature mapping layer is set to 1. At this time, the size of the first mapping information obtained is 4H*4W*128, and 4H*4W*128 can be understood as the size of the feature map is 4H*4W, and the number of channels is 128; the second sub-feature mapping layer performs feature mapping on the first mapping information to obtain second mapping information, and the size of the second mapping information is half of the size of the first mapping information. The step size in the second sub-feature mapping layer is set to 2, and the size of the second mapping information The second mapping information is feature mapped by the third sub-feature mapping layer to obtain third mapping information, and the size of the third mapping information is the same as that of the second mapping information. The step size of the convolution in the third sub-feature mapping layer is set to 1, and the size of the third mapping information is also 2H*2W*256, that is, the height, width and number of channels remain unchanged; the third mapping information is feature mapped by the fourth sub-feature mapping layer to obtain fourth mapping information, and the size of the fourth mapping information is half of the size of the third mapping information. The step size of the convolution in the fourth sub-feature mapping layer is set to 2, and the size of the fourth mapping information is H*W*512, that is, the mapping information corresponding to the target image output by the feature mapping layer is the fourth mapping information, and the height and width of the fourth mapping information are equivalent to 1 / 4 of the height and width of the target image. The constructed feature mapping layer includes multiple levels, which can extract the deep features of the target image. The extracted fourth mapping feature retains the character information in the target image to the maximum extent and removes the background information in the target image. It can be applied in scenes with more complex backgrounds, facilitates the subsequent recognition of characters based on the characteristic information of the characters, and improves the recognition accuracy.

[0063] Based on the above embodiments, Figure 5 A schematic diagram of a flow chart of an image processing method provided by an embodiment of the present disclosure, optionally, a feature matrix is ​​obtained based on mapping information and feature information through an adaptive clustering layer, specifically including the following: Figure 5 The following steps S510 to S540 are shown:

[0064] Optionally, the adaptive clustering layer includes an encoder, a first convolutional layer, and a second convolutional layer.

[0065] For example, see Figure 3 , Figure 3The adaptive clustering layer 330 includes an encoder, a first convolutional layer and a second convolutional layer. The encoder can specifically be an encoder of a Transformer structure, but the position encoder in the Transformer structure is not used. The position encoding is not used because the feature information input into the encoder contains clear feature point coordinate information, which is equivalent to a feature transformation. Secondly, the encoder output is also a vector. For example, if the input is 1024*512, the output is also 1024*512. The input 1024*512 is composed of the coordinate information of the 1024 feature points included in the feature information and the H*W*512 information included in the mapping information (the fourth mapping information).

[0066] S510: Obtain a first vector through an encoder based on the mapping information and the scaled feature information.

[0067] It can be understood that after the adaptive clustering layer receives the mapping information of the feature mapping layer and the feature information of the feature extraction layer, the encoder obtains the first vector based on the mapping information and the scaled feature information. The scaled feature information refers to the information after the feature information is reduced by 1 / 4. Since the mapping information is reduced by 1 / 4 compared to the height and width of the target image, the feature information obtained according to the target image also needs to be reduced by 1 / 4 accordingly to maintain the uniformity of the data and avoid errors. The size of the first vector output finally can be 1024*512.

[0068] S520. Perform feature transformation on the first vector through a first convolutional layer to obtain a second vector, where the second vector includes multiple feature points.

[0069] It can be understood that on the basis of the above S510, after the first vector is obtained by the encoder, the first convolution layer is used to perform feature transformation to obtain the second vector. The first convolution layer can be a 1*1 convolution layer. The first convolution layer can be understood as the number of compressed channels, that is, the number of compressed channels 512. The size of the second vector can be 1024*10, that is, the number of 512 channels in the first vector is compressed to 10 channels.

[0070] S530. Classify each feature point in the plurality of feature points in the second vector through a second convolutional layer to determine a category of the feature point, where the category of the feature point is a target line of the feature point in the plurality of lines of text.

[0071] It can be understood that, based on the above S520, after obtaining the second vector, the second convolution layer is used to classify each feature point in the multiple feature points included in the second vector to obtain the category of the feature point. The second convolution layer can be a 1*1 convolution layer. It can be understood that a feature point is equivalent to a character in the target image, or multiple feature points may be one character. Classifying feature points is to classify characters. The category refers to the number of text lines. The category of feature points is the target line of feature points in multiple lines of text, that is, to determine which text line the character corresponding to the feature point belongs to. The maximum number of lines in an image set above is 20 lines, so the number of categories is 20. For example, the target image includes 10 lines, and character A is in the second line of the target image. Then the category of the feature point corresponding to character A identified by the second convolution layer is the second line. By analogy, the category of each feature point is determined, that is, the text line where each character is located is determined. The number of text lines in the sample image can also be determined according to the category of the feature point. That is, if it is recognized that there are 5 categories of feature points, it can be determined that the sample image includes 5 lines of text.

[0072] S540: Obtain a feature matrix according to the category of the feature points, the mapping information, and the scaled feature information.

[0073] It can be understood that, based on the above S530, an N*512 feature matrix is ​​formed according to the category of each feature point obtained by the second convolutional layer, the mapping information output by the feature mapping layer, and the scaled feature information.

[0074] Optionally, the above S540 specifically includes: dividing the target feature points of the same category among multiple feature points into a group, and each group is separated by a preset character; determining the position of the target feature point in each group in the scaled feature information, and determining the vector of the target feature point in the mapping information according to the position of the target feature point; and forming a feature matrix according to the position of the target feature point and the vector of the target feature point.

[0075] Optionally, the above-mentioned determining the vector of the target feature point in the mapping information according to the position of the target feature point specifically includes: sorting the target feature points in each group according to the position of the target feature point; and determining the vector of the target feature point in the mapping information according to the position of the sorted target feature point.

[0076] It is understandable that the target feature points of the same category among multiple feature points are divided into a group, that is, all characters in the same text line are grouped together, and each group is separated by a preset character, which can be \n. According to the maximum number of text lines set above, it can be divided into 20 categories. If the sample image only includes 5 text lines, the category of the feature points will also be only 5 categories, and the remaining 15 categories can be automatically filled with empty characters. Then, the position information and feature mapping vector of each feature point in each group are determined. Specifically, in the same group, the position information of all feature points is determined in the scaled feature information, and the feature points are sorted in the group according to the position information. Then, according to the position information of the target feature point (current feature point), the feature mapping vector corresponding to the target feature point is determined in the mapping information, and then the feature mapping vector corresponding to the next feature point in the same group is determined. The maximum number of characters per line set above is 10. If there are no 10 characters in the same group (in a line), empty characters are automatically filled until the maximum number of characters 10 is reached. After the feature mapping vectors corresponding to each feature point in all the divided groups are determined, an N*512 feature matrix is ​​formed according to the categories, position information and feature mapping vectors of all the feature points.

[0077] An image processing method provided by an embodiment of the present disclosure can directly identify the information of feature points corresponding to all characters included in multiple lines of text. After determining the first vector through the encoder, feature transformation is continued based on the first convolution layer to reduce the number of channels, and then the category of each feature point is determined according to the second convolution layer. Only direct recognition of multiple lines of text involves the operation of determining the category of feature points to clarify the text line where each character corresponding to the feature point is located, and then the feature mapping vector of each feature point is determined to obtain a feature matrix. This method can effectively reduce error accumulation and improve recognition effect and recognition accuracy.

[0078] Figure 6 The image processing device provided by the embodiment of the present disclosure can execute the processing flow provided by the image processing method embodiment, such as Figure 6 As shown, the image processing device 600 includes:

[0079] An acquisition unit 610 is used to acquire a target image, where the target image includes multiple lines of text, and each line of the multiple lines of text includes at least one character;

[0080] The first recognition unit 620 is used to recognize characters in a target image by using a pre-trained text recognition model, wherein the text recognition model includes a feature mapping layer, a feature extraction layer, an adaptive clustering layer, and an output layer, wherein the feature mapping layer performs feature mapping on the target image to obtain mapping information corresponding to the target image, the feature extraction layer performs feature extraction on the target image to obtain feature information corresponding to the target image, the adaptive clustering layer obtains a feature matrix based on the mapping information and the feature information, and the output layer obtains a probability matrix corresponding to the target image based on the feature matrix;

[0081] The second recognition unit 630 is used to obtain the recognition result of the characters in the target image according to the probability matrix corresponding to the target image.

[0082] Optionally, the feature mapping layer in the first recognition unit 620 includes multiple sub-feature mapping layers, and the multiple sub-feature mapping layers are connected in sequence.

[0083] Optionally, the first recognition unit 620 performs feature mapping on the target image through a feature mapping layer to obtain mapping information corresponding to the target image, which is specifically used for:

[0084] Performing feature mapping on the target image through a first sub-feature mapping layer to obtain first mapping information, wherein the size of the first mapping information is the same as the size of the target image;

[0085] Performing feature mapping on the first mapping information through a second sub-feature mapping layer to obtain second mapping information, wherein the size of the second mapping information is half of the size of the first mapping information;

[0086] Performing feature mapping on the second mapping information through a third sub-feature mapping layer to obtain third mapping information, wherein the size of the third mapping information is the same as the size of the second mapping information;

[0087] The third mapping information is feature mapped through the fourth sub-feature mapping layer to obtain fourth mapping information, the size of the fourth mapping information is half of the size of the third mapping information, and the fourth mapping information is mapping information corresponding to the target image.

[0088] Optionally, the adaptive clustering layer in the first recognition unit 620 includes an encoder, a first convolutional layer and a second convolutional layer.

[0089] Optionally, the first recognition unit 620 obtains a feature matrix based on the mapping information and the feature information through an adaptive clustering layer, which is specifically used for:

[0090] Obtaining a first vector through an encoder based on the mapping information and the scaled feature information;

[0091] Performing feature transformation on the first vector through the first convolution layer to obtain a second vector, where the second vector includes a plurality of feature points;

[0092] Classifying each feature point in the plurality of feature points in the second vector through a second convolutional layer to determine a category of the feature point, where the category of the feature point is a target line of the feature point in the plurality of lines of text;

[0093] According to the category of feature points, mapping information and scaled feature information, a feature matrix is ​​obtained.

[0094] Optionally, the first recognition unit 620 obtains a feature matrix according to the category of the feature points, the mapping information and the scaled feature information, which is specifically used for:

[0095] Divide target feature points of the same category among multiple feature points into a group, and separate each group by a preset character;

[0096] Determine the position of the target feature point in each group in the scaled feature information, and determine the vector of the target feature point in the mapping information according to the position of the target feature point;

[0097] The feature matrix is ​​formed according to the position of the target feature point and the vector of the target feature point.

[0098] Optionally, the first recognition unit 620 determines the vector of the target feature point in the mapping information according to the position of the target feature point, specifically for:

[0099] Sort the target feature points in each group according to their positions;

[0100] According to the positions of the sorted target feature points, the vectors of the target feature points are determined in the mapping information.

[0101] Optionally, the first recognition unit 620 obtains a probability matrix corresponding to the target image based on the feature matrix through the output layer, which is specifically used for:

[0102] The probability matrix corresponding to the target image is obtained through the output layer based on the feature matrix and the preset dictionary, and the preset dictionary includes multiple characters.

[0103] Optionally, the apparatus 600 further includes a training unit, and the training unit is specifically used for:

[0104] Obtain a sample image and characters in the sample image;

[0105] Input the sample image into the constructed text recognition model to obtain the preset characters corresponding to the sample image;

[0106] Update the network parameters of the text recognition model based on the preset characters, characters in the sample image, and the preset loss function.

[0107] Optionally, the sample image and the characters in the sample image are obtained in the training unit, specifically for:

[0108] Get a sample image, which includes multiple lines of text;

[0109] The characters included in each line of the multiple lines of text are marked, and the characters included in each line of text in the sample image are determined. The multiple lines of text belonging to the same sample image are separated by preset characters.

[0110] Figure 6 The image processing device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0111] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0112] Exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method according to an embodiment of the present disclosure.

[0113] The exemplary embodiments of the present disclosure also provide a computer program product, including a computer program, wherein the computer program is used to cause the computer to execute the method according to the embodiments of the present disclosure when the computer program is executed by a processor of a computer.

[0114] refer to Figure 7 , a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0115] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0116] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0117] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. In some embodiments, the computing unit 701 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0118] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0119] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0123] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. An image processing method, characterized in that: include: Acquire a target image, wherein the target image includes multiple lines of text, and each line of text in the multiple lines of text includes at least one character; The characters in the target image are recognized by a pre-trained text recognition model, the text recognition model includes a feature mapping layer, a feature extraction layer, an adaptive clustering layer and an output layer, wherein the feature mapping layer performs feature mapping on the target image to obtain mapping information corresponding to the target image, the feature extraction layer performs feature extraction on the target image to obtain feature information corresponding to the target image, and the adaptive clustering layer obtains a feature matrix based on the mapping information and the feature information, including: obtaining a first vector based on the mapping information and the scaled feature information by an encoder included in the adaptive clustering layer; performing feature transformation on the first vector by a first convolutional layer included in the adaptive clustering layer to obtain a second vector, the second vector including a plurality of feature points; classifying each feature point in the plurality of feature points in the second vector by a second convolutional layer included in the adaptive clustering layer to determine the category of the feature point, the category of the feature point being the target line of the feature point in the plurality of lines of text; obtaining a feature matrix according to the category of the feature point, the mapping information and the scaled feature information, and obtaining a probability matrix corresponding to the target image based on the feature matrix by the output layer; According to the probability matrix corresponding to the target image, the recognition result of the character in the target image is obtained.

2. The method according to claim 1, characterized in that The feature mapping layer includes a plurality of sub-feature mapping layers, and the plurality of sub-feature mapping layers are sequentially connected; The performing feature mapping on the target image through the feature mapping layer to obtain mapping information corresponding to the target image includes: Performing feature mapping on the target image through a first sub-feature mapping layer to obtain first mapping information, wherein the size of the first mapping information is the same as the size of the target image; Performing feature mapping on the first mapping information through a second sub-feature mapping layer to obtain second mapping information, wherein the size of the second mapping information is half of the size of the first mapping information; Performing feature mapping on the second mapping information through a third sub-feature mapping layer to obtain third mapping information, wherein the size of the third mapping information is the same as the size of the second mapping information; The third mapping information is feature mapped through a fourth sub-feature mapping layer to obtain fourth mapping information, where the size of the fourth mapping information is half of the size of the third mapping information, and the fourth mapping information is mapping information corresponding to the target image.

3. The method according to claim 1, characterized in that The step of obtaining a feature matrix according to the category of the feature points, the mapping information and the scaled feature information includes: Dividing the target feature points of the same category among the multiple feature points into a group, wherein each group is separated by a preset character; Determine the position of the target feature point in each group in the scaled feature information, and determine the vector of the target feature point in the mapping information according to the position of the target feature point; A feature matrix is ​​formed according to the positions of the target feature points and the vectors of the target feature points.

4. The method according to claim 3, characterized in that The determining the vector of the target feature point in the mapping information according to the position of the target feature point comprises: Sorting the target feature points in each group according to their positions; According to the sorted positions of the target feature points, vectors of the target feature points are determined in the mapping information.

5. The method according to claim 1, characterized in that The obtaining, through the output layer based on the feature matrix, a probability matrix corresponding to the target image includes: A probability matrix corresponding to the target image is obtained through the output layer based on the feature matrix and a preset dictionary, wherein the preset dictionary includes a plurality of characters.

6. The method according to claim 1, characterized in that The method further comprises: Acquire a sample image and characters in the sample image; Inputting the sample image into the constructed text recognition model to obtain the preset characters corresponding to the sample image; The network parameters of the text recognition model are updated according to the preset characters, the characters in the sample image and the preset loss function.

7. The method according to claim 6, characterized in that The obtaining of the sample image and the characters in the sample image comprises: Acquire a sample image, wherein the sample image includes multiple lines of text; The characters included in each line of the multiple lines of text are marked, and the characters included in each line of text in the sample image are determined, and the multiple lines of text belonging to the same sample image are separated by preset characters.

8. An image processing device, characterized in that: include: An acquisition unit, configured to acquire a target image, wherein the target image includes multiple lines of text, each line of the multiple lines of text includes at least one character; A first recognition unit is used to recognize characters in the target image through a pre-trained text recognition model, wherein the text recognition model includes a feature mapping layer, a feature extraction layer, an adaptive clustering layer and an output layer, wherein the feature mapping layer performs feature mapping on the target image to obtain mapping information corresponding to the target image, the feature extraction layer performs feature extraction on the target image to obtain feature information corresponding to the target image, and the adaptive clustering layer obtains a feature matrix based on the mapping information and the feature information, including: obtaining a first vector based on the mapping information and the scaled feature information through an encoder included in the adaptive clustering layer; performing feature transformation on the first vector through a first convolutional layer included in the adaptive clustering layer to obtain a second vector, wherein the second vector includes a plurality of feature points; classifying each feature point in the plurality of feature points in the second vector through a second convolutional layer included in the adaptive clustering layer to determine a category of the feature point, wherein the category of the feature point is a target line of the feature point in the plurality of lines of text; obtaining a feature matrix based on the category of the feature point, the mapping information and the scaled feature information, and obtaining a probability matrix corresponding to the target image through the output layer based on the feature matrix; The second recognition unit is used to obtain the recognition result of the character in the target image according to the probability matrix corresponding to the target image.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images