Image text recognition method, device, equipment and storage medium

Abstract

The present application discloses a method, apparatus, device and storage medium for image text recognition, and relates to image processing technology in the field of cloud computing. The specific implementation scheme is: obtaining an image to be processed, in which there is at least one text line, processing each text line in the image to be processed, and obtaining a synthetic coding vector corresponding to each single word in each text line, the synthetic coding vector carries semantic information and position information, and according to the semantic information and position information carried by the synthetic coding vector corresponding to each single word in each text line, determining the text recognition result in the image to be processed. In this technical scheme, not only the semantic features of the text are utilized, but also the position information of the text on the image is integrated, which can accurately distinguish adjacent fields with small pixel spacing in the image, thereby improving the accuracy of text recognition in the image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image text recognition method, device, equipment and storage medium in image processing technology. Background Art

[0002] Optical character recognition (OCR) is the process of analyzing and identifying textual images to obtain text and layout information. In practical applications, after image recognition, it is often necessary to segment the continuous text within the image representation layer at the semantic level to distinguish the semantic differences between different fields.

[0003] However, the text detection process of existing OCR methods relies entirely on the semantic information of the image and cannot distinguish adjacent fields with small pixel spacing in the image, resulting in low recognition accuracy. Summary of the Invention

[0004] The embodiments of the present application provide an image text recognition method, apparatus, device and storage medium for solving the problem of low recognition accuracy caused by the inability of existing OCR methods to distinguish adjacent fields with very small pixel spacing in an image.

[0005] In a first aspect, the present application provides an image text recognition method, comprising:

[0006] Acquire an image to be processed, wherein the image to be processed contains at least one text line;

[0007] Processing each text line in the image to be processed to obtain a synthetic coding vector corresponding to each word in each text line, wherein the synthetic coding vector carries semantic information and position information;

[0008] The text recognition result in the image to be processed is determined based on the semantic information and position information carried by the synthetic coding vector corresponding to each single word in each text line.

[0009] In this embodiment, when performing image text recognition, not only the semantic features of the text are utilized, but also the position information of the text on the image is integrated, which can accurately distinguish adjacent fields with small pixel spacing in the image, thereby improving the accuracy of text recognition in the image.

[0010] In a possible design of the first aspect, processing each text line in the image to be processed to obtain a synthetic coding vector corresponding to each word in each text line includes:

[0011] Analyzing the semantics of each word in each text line in the image to be processed to obtain a semantic encoding vector corresponding to each word in each text line;

[0012] Determining coordinate information of each word in each text line in the image to be processed;

[0013] Encoding and latitude transforming the coordinate information of each word to obtain a position encoding vector of each word, wherein the position encoding vector has the same dimension as the semantic encoding vector;

[0014] The semantic encoding vector and position encoding vector of each word are fused to obtain the synthetic encoding vector of each word.

[0015] In this embodiment, the synthetic coding vector of each single word is obtained by fusing the semantic coding vector and position coding vector of each single word. It carries accurate semantic information and position information. Therefore, it can accurately divide different fields in a certain type of area (such as a table area) that may not have obvious semantic differences (such as multiple consecutive columns of numbers) but belong to different fields, thereby improving the accuracy of subsequent image text recognition.

[0016] Optionally, the semantics of each word in each text line in the image to be processed are analyzed to obtain a semantic coding vector corresponding to each word in each text line, including:

[0017] For each single word in the image to be processed, semantic information of all single words in the text line to which the single word belongs is weightedly fused to obtain a semantic encoding vector of the single word.

[0018] In this embodiment, the semantic coding vector of each word is obtained by weighted fusion, which improves the accuracy of the semantic coding vector.

[0019] Optionally, encoding and latitude transforming the coordinate information of each word to obtain a position coding vector of each word includes:

[0020] Determining a plurality of pixel points representing each word in the image to be processed;

[0021] Determine the position vector of each word based on the coordinate information of multiple pixel points corresponding to each word;

[0022] According to the dimension of the semantic coding vector corresponding to each single word, the position vector of each single word is transformed in dimension to obtain a position coding vector with the same dimension as the semantic coding vector.

[0023] In this embodiment, the position vector of each word is determined using multiple pixel points of each word, and then converted into a position coding vector with the same dimension as the semantic coding vector, which makes subsequent vector fusion possible.

[0024] In another possible design of the first aspect, determining the text recognition result in the image to be processed based on the semantic information and position information carried by the synthesized coding vector corresponding to each word in each text line includes:

[0025] Inputting the synthesized encoding vector corresponding to each word in each text line into a preset information analysis network to determine the context information of each word, wherein the preset information analysis network is used to analyze the context information of each word;

[0026] Determine the category label of each word based on the context information of each word;

[0027] The context information of each word in each text line and the category label to which each word belongs are input into the discriminant model to obtain the text recognition result in the image to be processed.

[0028] In this embodiment, by determining the context information and category of each word and then using the discriminant model to classify the category labels, the fields in the text line can be effectively segmented, laying the foundation for subsequent accurate text recognition results.

[0029] In another possible design of the first aspect, before processing each text line in the image to be processed to obtain a synthesized encoding vector corresponding to each word in each text line, the method further includes:

[0030] Optical character recognition is performed on the image to be processed to determine the text lines in the image to be processed and the position information of each word in each text line.

[0031] In another possible design of the first aspect, the method further includes:

[0032] Determining the range of each field in each text line according to the text recognition result in the image to be processed, each field including a plurality of words associated with each other in contextual semantic information;

[0033] The actual meaning of the text in the image to be processed is determined according to the contextual semantic information of all words in the same field in each text line.

[0034] In this embodiment, each text line is segmented according to the range of each field, thereby achieving accurate sentence segmentation to determine the actual meaning of the text in the image to be processed, which greatly facilitates the subsequent OCR text structuring work.

[0035] In a second aspect, the present application provides an image text recognition device, comprising: an acquisition module, a processing module, and a determination module;

[0036] The acquisition module is used to acquire an image to be processed, wherein the image to be processed contains at least one text line;

[0037] The processing module is configured to process each text line in the image to be processed to obtain a synthetic coding vector corresponding to each word in each text line, wherein the synthetic coding vector carries semantic information and position information;

[0038] The determination module is used to determine the text recognition result in the image to be processed based on the semantic information and position information carried by the synthetic coding vector corresponding to each single word in each text line.

[0039] In a possible design of the second aspect, the processing module is specifically used to analyze the semantics of each word in each text line in the image to be processed, obtain the semantic coding vector corresponding to each word in each text line, determine the coordinate information of each word in each text line in the image to be processed, encode and latitude transform the coordinate information of each word to obtain the position coding vector of each word, the position coding vector has the same dimension as the semantic coding vector, and fuse the semantic coding vector and position coding vector of each word to obtain a synthetic coding vector for each word.

[0040] Optionally, the processing module is configured to analyze the semantics of each word in each text line in the image to be processed to obtain a semantic encoding vector corresponding to each word in each text line, specifically:

[0041] The processing module is specifically configured to weightedly fuse the semantic information of all words in the text line to which each word in the image to be processed belongs, so as to obtain a semantic coding vector of the word.

[0042] Optionally, the processing module is used to encode and latitude-transform the coordinate information of each word to obtain a position coding vector of each word, specifically:

[0043] The processing module is specifically used to determine multiple pixel points representing each single word in the image to be processed, determine the position vector of each single word based on the coordinate information of the multiple pixel points corresponding to each single word, and perform dimension transformation on the position vector of each single word based on the dimension of the semantic coding vector corresponding to each single word to obtain a position coding vector with the same dimension as the semantic coding vector.

[0044] In another possible design of the second aspect, the determination module is specifically used to input the synthetic coding vector corresponding to each single word in each text line into a preset information analysis network to determine the context information of each single word. The preset information analysis network is used to analyze the context information of each single word, and determine the category label to which each single word belongs based on the context information of each single word. The context information of each single word in each text line and the category label to which each single word belongs are input into the discriminant model to obtain the text recognition result in the image to be processed.

[0045] In another possible design of the second aspect, the processing module is further used to perform optical character recognition on the image to be processed before processing each text line in the image to be processed to obtain the synthetic coding vector corresponding to each single word in each text line, and determine the position information of the text lines in the image to be processed and each single word in each text line.

[0046] In another possible design of the second aspect, the determination module is further used to determine the scope of each field in each text line based on the text recognition results in the image to be processed, each field includes multiple words that are mutually related by contextual semantic information, and the actual meaning of the text in the image to be processed is determined based on the contextual semantic information of all words in the same field in each text line.

[0047] The device provided in the second aspect of this application can be used to execute the method provided in the first aspect. Its implementation principle and technical effects are similar and will not be repeated here.

[0048] In a third aspect, the present application provides an electronic device, comprising:

[0049] at least one processor; and

[0050] a memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect and various possible designs of the first aspect.

[0052] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect and each possible design of the first aspect.

[0053] In a fifth aspect, the present application provides an image text recognition method, comprising:

[0054] Determine a synthetic encoding vector for each word in at least one text line of the image to be processed, wherein the synthetic encoding vector carries semantic information and position information;

[0055] The synthetic coding vector corresponding to each single word in each text line is processed to obtain the text recognition result in the image to be processed.

[0056] One embodiment of the aforementioned application has the following advantages or beneficial effects: by obtaining an image to be processed containing at least one text line, processing each text line in the image to be processed, and obtaining a synthetic coding vector corresponding to each individual word in each text line. The synthetic coding vector carries semantic information and positional information. Based on the semantic information and positional information carried by the synthetic coding vector corresponding to each individual word in each text line, the text recognition result in the image to be processed is determined. This technical solution not only utilizes the semantic features of the text but also incorporates the positional information of the text in the image. It can accurately distinguish adjacent fields in the image with very close pixel spacing, thereby improving the accuracy of text recognition in the image.

[0057] Other effects of the above optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present application.

[0059] Figure 1 1 is a flow chart of an image text recognition method according to the first embodiment of the present application;

[0060] Figure 2 2 is a flow chart of an image text recognition method according to the second embodiment of the present application;

[0061] Figure 3 A schematic diagram of processing an image to be processed to obtain a synthetic coding vector for each single word;

[0062] Figure 4 1 is a flow chart of an image text recognition method according to the third embodiment of the present application;

[0063] Figure 5 Schematic diagram of processing a single word in a text line;

[0064] Figure 6 4 is a flowchart of an image text recognition method according to the fourth embodiment of the present application;

[0065] Figure 7 This is a schematic diagram of a hospital's patient expense list;

[0066] Figure 8 for Figure 7 Schematic diagram of text test results in the patient's billing statement shown;

[0067] Figure 9This is a block diagram of an electronic device performing image and text recognition in an embodiment of the present application;

[0068] Figure 10 for Figure 7 The schematic diagram of the recognition result of the patient's expense list is shown;

[0069] Figure 11 A schematic diagram of the structure of an image and text recognition device provided in an embodiment of the present application;

[0070] Figure 12 It is a block diagram of an electronic device used to implement the image text recognition method of the embodiment of the present application. DETAILED DESCRIPTION

[0071] The following description of exemplary embodiments of the present application is made in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0072] Optical character recognition (OCR) refers to the process of analyzing, identifying, and processing image files of text materials to obtain text and layout information. In other words, the text in the image is identified and returned as text. A typical OCR technical solution can be divided into two parts: text detection and text recognition. Text detection is the detection of the location, range, and layout of text in the image, and usually also includes layout analysis and text line detection. Text detection mainly determines where in the image there is text and the extent of the text. Text recognition, based on text detection, identifies the text content and converts the text information in the image into text information. Text recognition mainly determines the identity of each detected character.

[0073] For text detection tasks, the current main method is to apply image detection methods to select text areas in images. A commonly used algorithm is the Faster R-CNN algorithm, which uses region proposal networks (RPN) to assist in generating samples. The algorithm structure is divided into two parts. The RPN is first used to determine whether the candidate box is the target box, and then the type of target box is determined by the multi-task loss function of classification and positioning. The entire network process can share the feature information extracted by the convolutional neural network, saving computational costs and solving the problem of the Fast R-CNN algorithm's slow speed in generating candidate boxes for positive and negative samples. At the same time, it avoids the reduction in algorithm accuracy caused by excessive candidate box extraction. For text detection in restricted scenes, the Faster R-CNN algorithm performs relatively well and can determine text areas of different granularities through multiple detections.

[0074] However, although the above method can solve the problem of text detection in OCR to a certain extent, since the detection method is completely dependent on the semantic feature information of the image, it can only distinguish text lines with obvious differences in image features. However, in actual applications, after image recognition, it is usually necessary to segment the continuous text in the image representation layer at the text semantic level to distinguish the semantic differences between different fields. To address this problem, related technologies mainly rely on the characteristics of text expression to manually extract rules to segment the fields. Since the process of formulating manual rules is cumbersome and complex, it is time-consuming and has poor flexibility, and there may be problems of mutual influence between different strategies.

[0075] In summary, the text detection process of the OCR method in the prior art is completely dependent on the semantic information of the image, and is unable to distinguish adjacent fields with very small pixel spacing in the image, resulting in low recognition accuracy. Therefore, the embodiment of the present application provides an image text recognition method, which obtains an image to be processed, wherein there is at least one text line in the image to be processed, processes each text line in the image to be processed, and obtains a synthetic coding vector corresponding to each single word in each text line. The synthetic coding vector carries semantic information and position information, and the text recognition result in the image to be processed is determined based on the semantic information and position information carried by the synthetic coding vector corresponding to each single word in each text line. In this technical solution, not only the semantic features of the text are utilized, but also the position information of the text on the image is integrated, which can accurately distinguish adjacent fields with very small pixel spacing in the image, thereby improving the accuracy of text recognition in the image.

[0076] It is understood that the execution subject of the embodiments of the present application can be an electronic device, such as a terminal device such as a computer or tablet computer, or a server, such as a backend processing platform. Therefore, the present embodiment will be explained by collectively referring to terminal devices and servers as electronic devices. Whether the electronic device is specifically a terminal device or a server can be determined in actual circumstances.

[0077] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0078] Figure 1 FIG. 1 is a flow chart of the image text recognition method according to the first embodiment of the present application. Figure 1 As shown, the method may include the following steps:

[0079] S101: Acquire an image to be processed, wherein the image to be processed contains at least one text line.

[0080] In practical applications, image and text recognition is widely used in virtual reality, human-computer interaction, bill recognition, industrial automation, and other fields. For example, by recognizing bills, it is possible to determine the information contained in the bills, helping relevant personnel to collect and process the bills electronically.

[0081] In this embodiment, when an electronic device needs to perform an image text recognition operation, it first obtains an image to be processed containing text. The image to be processed can be an image with text pre-installed in the electronic device, or it can be an image to be processed independently of the electronic device. This embodiment of the application does not limit the method for obtaining the image to be processed, and the method can be determined based on actual needs.

[0082] It is understandable that there needs to be at least one text line in the image to be processed so that the electronic device can accurately identify each field in each text line using the image text recognition method of this embodiment, thereby determining the actual meaning of the text in the image.

[0083] S102 , processing each text line in the image to be processed to obtain a synthetic coding vector corresponding to each word in each text line.

[0084] The synthesized coding vector carries semantic information and position information.

[0085] Exemplarily, in this embodiment, a self-attention & positional embedding layer can be provided in the electronic device. Specifically, the self-attention mechanism of the self-attention & positional embedding layer is used to process each text line in the processed image, and the semantic information of each word is determined by encoding the contextual semantic information of each word. The positional encoding mechanism of the self-attention & positional embedding layer is used to encode the position information of each word in each text line to determine the position information of each word. Finally, the semantic information and positional information of each word are fused to obtain a synthetic encoding vector corresponding to each word in each text line.

[0086] The implementation principle of this step can be found in the following Figure 2 The description in the illustrated embodiment will not be repeated here.

[0087] It is understood that, in the embodiment of the present application, before this step, the method may further include the following steps:

[0088] Perform optical character recognition on the image to be processed to determine the position information of the text lines in the image to be processed and each word in each text line.

[0089] In practical applications, after obtaining an image to be processed, it is usually necessary to first determine whether there are words in the image to be processed and the location of each word if there are words in the image to be processed. In this embodiment, text detection and word recognition can be performed on the image to be processed using an optical character recognition (ORC) method.

[0090] Specifically, the image to be processed is first scanned to determine the text lines in the image to be processed and the location, range and layout of the text lines. Then, each text line is detected to determine the specific meaning of each word.

[0091] S103 , determining a text recognition result in the image to be processed based on the semantic information and position information carried by the synthetic coding vector corresponding to each single word in each text line.

[0092] In this embodiment, because the synthetic code vector corresponding to each word in a text line incorporates both semantic and positional information, the synthetic code vector output at each position in the processed image simultaneously represents both semantic and positional information. When this synthetic code vector is input into a discriminant model for entity sequence label classification, an accurate sequence label classification result can be obtained. Based on the resulting sequence label splitting results, the coordinate ranges of all words in the same entity field are aggregated to determine the text recognition result in the processed image.

[0093] The implementation principle of this step can be found in the following Figure 4 The description in the illustrated embodiment will not be repeated here.

[0094] The image text recognition method provided by the embodiments of the present application obtains a to-be-processed image containing at least one text line, processes each text line in the to-be-processed image, and obtains a synthetic coding vector corresponding to each individual word in each text line. The synthetic coding vector carries semantic information and positional information. The text recognition result in the to-be-processed image is determined based on the semantic information and positional information carried by the synthetic coding vector corresponding to each individual word in each text line. This technical solution not only utilizes the semantic features of the text but also incorporates the positional information of the text in the image. It can accurately distinguish adjacent fields in the image with very small pixel spacing, thereby improving the accuracy of text recognition in the image.

[0095] For example, based on the above embodiment, Figure 2 FIG. 1 is a flow chart of an image text recognition method according to the second embodiment of the present application. Figure 2 As shown, in this embodiment, the above S102 can be implemented by the following steps:

[0096] S201 , analyzing the semantics of each word in each text line in the image to be processed to obtain a semantic coding vector corresponding to each word in each text line.

[0097] In this embodiment, for each text line in the image to be processed, semantic enhancement is first performed on each word therein to obtain an enhanced semantic vector. Then, each word is taken as an object, for example, a query in a database, and the semantic information of all words in the text line where the word is located is weightedly fused to obtain the semantic encoding vector corresponding to each word.

[0098] For example, for each word, the semantic encoding vector obtained after weighted fusion can be represented by a 768-dimensional vector.

[0099] Therefore, in this embodiment, this step can be summarized as being implemented through the following steps: for each word in the image to be processed, weighted fusion of the semantic information of all words in the text line to which the word belongs is performed to obtain the semantic encoding vector of the word.

[0100] S202: Determine the coordinate information of each word in each text line in the image to be processed.

[0101] In this embodiment, after the image to be processed undergoes optical character recognition, the position coordinates of each word in each text line in the image to be processed can be obtained, thereby determining the coordinate information of each word in each text line in the image to be processed.

[0102] S203 , encoding and latitude transforming the coordinate information of each word to obtain a position coding vector of each word.

[0103] The position encoding vector has the same dimension as the semantic encoding vector.

[0104] In this embodiment, after obtaining the coordinate information of each word in the image to be processed, multiple pixel points representing the word can be determined, and the coordinate information of the multiple pixel points is used to form a position vector to represent the word.

[0105] For example, the upper left corner, center point and lower right corner point of each word are used as the pixel points of the word. Therefore, a 6-dimensional vector can be formed using the coordinates of the upper left corner point, the coordinates of the center point and the coordinates of the lower right corner point of each word to obtain the position vector of the word.

[0106] Furthermore, in this embodiment, the position vector of each word can be transformed to obtain a position encoding vector with the same dimensions as the semantic encoding vector. Exemplarily, a fully connected layer is used to map the position vector of the word to a 768-dimensional vector, making it the same dimension as the semantic encoding vector. This allows the position encoding vector and the semantic encoding vector to be added together to obtain a composite encoding vector for each word.

[0107] Therefore, in the embodiment of the present application, S203 can be implemented by the following steps:

[0108] A1: Determine multiple pixel points representing each word in the image to be processed.

[0109] Optionally, in the image to be processed, each word occupies a certain space, that is, each word is composed of multiple pixels. In this embodiment, in order to accurately represent the position information of the word, the coordinate information of the multiple pixels that make up the word can be used to represent it. Specifically, since the center point, upper left corner point, lower right corner point, upper right corner point, lower left corner point, etc. of each word are all key pixel points of the word, it is possible to select from the above-mentioned center point, upper left corner point, lower right corner point, upper right corner point, and lower left corner point.

[0110] A2: Determine the position vector of each word based on the coordinate information of multiple pixel points corresponding to each word.

[0111] For example, to simplify processing complexity and accurately characterize the position of a word, in this embodiment, the center point, upper left corner, and lower right corner of each word can be used as key pixels. Thus, the relative distances between each two pixels in the three pixels can be calculated using the coordinates of the center point, upper left corner, and lower right corner of each word, thereby facilitating the formation of a 6-dimensional position vector from the three sets of relative distances. This 6-dimensional position vector is the position vector of the word.

[0112] A3: According to the dimensions of the semantic encoding vectors corresponding to the individual words, the position vectors of the individual words are transformed to obtain position encoding vectors with the same dimensions as the semantic encoding vectors.

[0113] In this embodiment, since the semantic coding vector obtained after semantic analysis of each word in the text line is a high-dimensional vector, in order to fuse the semantic information and position information of the word, in this embodiment, it is necessary to perform dimensionality transformation on the position vector of each word according to the dimension of the semantic coding vector corresponding to each word, so as to transform the position vector of each word into a position coding vector with the same dimension as the semantic coding vector.

[0114] S204: Fusing the semantic encoding vector and the position encoding vector of each word to obtain a composite encoding vector for each word.

[0115] In this embodiment, after obtaining the semantic coding vector and position coding vector corresponding to each word, since the position coding vector and the semantic coding vector of each word have the same dimension, the semantic coding vector and the position coding vector of each word are fused through the principle of superposition at the same position, and the fused vector is used as the synthetic coding vector of each word.

[0116] In this embodiment, since the semantic coding vector of each word is obtained by weighted fusion of the semantic information of all words in the text line to which the word belongs, and the position coding vector of each word is obtained by encoding and latitude transforming the coordinate information of the word, the synthetic coding vector of each word carries accurate semantic information and position information.

[0117] For example, Figure 3 Schematic diagram of processing the image to be processed to obtain the synthetic coding vector of each word. Figure 3As shown, in this embodiment, optical character recognition is first performed on the image to be processed to determine at least one text line in the image to be processed, and then semantic analysis is performed on each word in each text line to obtain the semantic coding vector corresponding to each word, and the position coding vector of each word is obtained based on the coordinate information of each word, and finally the semantic coding vector and position coding vector of each word are fused to obtain a composite coding vector for each word.

[0118] The image text recognition method provided by the embodiment of the present application analyzes the semantics of each word in each text line in the image to be processed to obtain the semantic coding vector corresponding to each word in each text line, then determines the coordinate information of each word in each text line in the image to be processed, encodes and transforms the coordinate information of each word, obtains the position coding vector of each word, and the position coding vector has the same dimension as the semantic coding vector, and finally fuses the semantic coding vector and the position coding vector of each word to obtain the synthetic coding vector of each word. In this technical solution, the synthetic coding vector of each word is obtained by fusing the semantic coding vector and the position coding vector of each word, which carries accurate semantic information and position information. Therefore, it can accurately divide different fields in a certain type of area (such as a table area) that may not have obvious semantic differences (such as multiple consecutive digital columns), but belong to different fields, thereby improving the accuracy of subsequent image text recognition.

[0119] For example, based on the above embodiment, Figure 4 FIG. 1 is a flow chart of an image text recognition method according to the third embodiment of the present application. Figure 4 As shown, in this embodiment, the above S103 can be implemented by the following steps:

[0120] S401: Input the synthesized coding vector corresponding to each word in each text line into a preset information analysis network to determine the context information of each word.

[0121] The preset information analysis network is used to analyze the context information of each word.

[0122] Exemplarily, the preset information analysis network can be implemented by a bidirectional long short-term memory (Bi-LSTM) network. The Bi-LSTM network includes a forward LSTM network and a backward LSTM network. The forward LSTM network is used to analyze the context of the current word, and the backward LSTM network is used to analyze the context of the current word.

[0123] Specifically, in this embodiment, the context information of the current single character is encoded by the forward LSTM and the backward LSTM respectively, and the context information of each single character can be determined by concatenating the output results of the forward LSTM and the backward LSTM.

[0124] Specifically, for the single character x i (i is a positive integer), the semantic recognition results of the previous (i - 1) single characters of the single character x i are used as the context information of the single character x i , and the semantic recognition results of the single characters after the single character x i are used as the context information of the single character x i . Optionally, the single character x i encodes the context information of the single character x i through the forward LSTM, and the output result of the context of the single character x i is denoted as Similarly, the single character x i encodes the context information of the single character x i through the backward LSTM, and the output result of the context of the single character x i is Concatenate the output result of the context of the single character x i and the output result of the context at the vector level to obtain the bidirectional LSTM output result p i of the single character x i , and the dimension of this output result p i is twice that of the output result of the context [ and the output result of the context and the output result of the context .

[0125] Exemplarily, Figure 5 is a schematic diagram of processing single characters in a text line. As Figure 5 shown, in this embodiment, "China is very big" detected by optical character recognition is used as the text line to be processed, and the single characters and their positions in this text line are referred to Figure 5 shown. Specifically, the synthetic coding vector corresponding to "中" is x1, the synthetic coding vector corresponding to "国" is x2, the synthetic coding vector corresponding to "很" is x3, and the synthetic coding vector corresponding to "大" is x4. Input the synthetic coding vector x1 into the bidirectional LSTM, and the output result of the forward LSTM is The output result of the backward LSTM is Add the output result and the output result to obtain the output result of the bidirectional LSTM as p1. That is, the vector corresponding to the context information of the single character "中" is p1.

[0126] Similarly, the vector corresponding to the context information of the single character "country" is p2, the vector corresponding to the context information of "very" is p3, and the vector corresponding to the context information of "big" is p4. In this embodiment, the processing process for other single characters is similar and will not be elaborated here.

[0127] S402. Determine the category labels to which each single character belongs according to the context information of each single character.

[0128] In this embodiment, the vector corresponding to the context information of each single character can be linearly transformed through one layer to obtain the score of the single character belonging to each category label. That is, through the linear transformation layer, the output result of the bidirectional LSTM of each single character can be transformed into a vector with the same dimension as the number of category labels, so as to determine the category labels to which each single character belongs.

[0129] S403. Input the context information of each single character and the category labels to which each single character belongs in each text line into the discriminant model to obtain the text recognition result in the to-be-processed image.

[0130] Optionally, the discriminant model is a conditional random fields (CRF) layer. In this embodiment, according to the context information of each single character output by the bidirectional LSTM, the category labels to which each single character belongs, and combined with the transition score matrix of the conditional random fields, an optimal entity label sequence can be obtained.

[0131] In this embodiment, the main function of the CRF is to learn the state transition score matrix, and some constraints can be added to ensure that the final prediction result is valid. For example: the beginning of a sentence should be "B-" or "O", rather than "I-" or "E-"; "O I-label" is incorrect, and the beginning of a named entity should be "B-" rather than "I-", etc.

[0132] It can be understood that the embodiment of the present application can perform text structuring tasks on the table area, and obtain the text and coordinate information in at least one text line through general OCR detection and recognition.

[0133] For the text in the same line, this embodiment performs secondary processing on it to split multiple fields with very small pixel distances. Using the deep learning method, the field splitting task is transformed into a common named entity recognition task in natural language processing. Each column of fields is regarded as an entity, and in the way of "B-entity name" (which represents the start word of a certain entity), "I-entity name" (which represents the middle word of a certain entity), "E-entity name" (which represents the end word of a certain entity), "S-entity name" (which represents the single word entity), and "O" (which represents that the word does not form an entity), the prediction output of sequence tags is made for the table text in the same line. According to the sequence tag prediction result, the text range belonging to the same entity can be known. By aggregating the single-word coordinates of the same entity, the text detection box corresponding to the field can be regenerated.

[0134] Exemplarily, referring to Figure 5 As shown, in the text recognition result corresponding to the above text line "China is very big", "中" corresponds to "B-LOC", "国" corresponds to "I-LOC", and both "很" and "大" correspond to "O".

[0135] The image text recognition method provided by the embodiment of the present application inputs the synthetic coding vectors corresponding to each single word in each text line into a preset information analysis network to determine the context information of each single word. According to the context information of each single word, the category label to which each single word belongs is determined. The context information of each single word and the category label to which each single word belongs in each text line are input into a discriminant model to obtain the text recognition result in the to-be-processed image. In this technical solution, by determining the context information and the category to which each single word belongs, and then using the discriminant model for category label classification, the fields in the text line can be effectively split, laying a foundation for obtaining an accurate text recognition result subsequently.

[0136] Further, in the embodiment of the present application, Figure 6 is a flowchart of the image text recognition method provided according to the fourth embodiment of the present application. As Figure 6 shown, the method may further include the following steps:

[0137] S601. According to the text recognition result in the to-be-processed image, determine the range of each field in each text line, and each field includes multiple single words with interrelated context semantic information.

[0138] Exemplarily, this step may be executed after S103 above, that is, according to the obtained text recognition result in the to-be-processed image, the fields in each text line are divided. Exemplarily, the range of each field can be determined based on the category entity label carried by each single word, so that the context semantic information of the multiple single words included in each field is related.

[0139] S602: Determine the actual meaning of the text in the image to be processed based on the contextual semantic information of all words in the same field in each text line.

[0140] In this embodiment, each text line can be segmented according to the range of each field, thereby achieving accurate sentence segmentation to determine the actual meaning of the text in the image to be processed, which greatly facilitates the subsequent OCR text structuring work.

[0141] The image text recognition method of the embodiment of the present application determines the scope of each field in each text line based on the text recognition results in the image to be processed. Each field includes multiple words that are mutually related in contextual semantic information. Based on the contextual semantic information of all words in the same field in each text line, the actual meaning of the text in the image to be processed is determined, which can assist in performing the structuring task of text in subsequent images.

[0142] In summary, after obtaining the OCR detection and recognition results, the embodiment of the present application performs a second processing to encode each word in the text line so that each word fully integrates the contextual semantic information, and by encoding the position information of the detection frame of each word and fusing it with the semantic information, the output vector of each position simultaneously represents the semantic information and the position information. A bidirectional LSTM network is connected to the upper layer to encode the context information, and a sequence label output is made by connecting to CRF. The range of each entity field can be obtained according to the sequence label result of the entity type. Finally, the coordinate range of all words in the same entity field is re-aggregated to generate a new detection frame, completing the field segmentation task based on text semantics and position encoding.

[0143] Below, the embodiment of the present application uses an example to illustrate the text structuring task for the table area of ​​a detailed list issued by a hospital. Since the table area of ​​an image in a real scene often has the problem of very small spacing between multiple columns, and since the text detection process completely relies on the semantic information of the image, it is impossible to distinguish continuous text that has different text semantics but no obvious features in the image position, which makes it difficult to distinguish multiple fields with very small pixel spacing during the OCR detection stage. The method of the embodiment of the present application can segment the OCR results according to the text semantics and the position of the text on the image, which greatly facilitates the subsequent OCR text structuring work.

[0144] For example, Figure 7 This is a diagram of a hospital's patient expense list. Figure 7As shown, for the area outlined by the thick dashed line, since the distance between adjacent columns is relatively close and there are no obvious features in the image to distinguish different fields, therefore, exemplarily, existing OCRs usually detect the two fields of "16 layers unknown" and "block" as one field, and detect the two fields of "2.1100" and "4" as one field. This embodiment mainly performs secondary processing on the OCR detection and recognition results to achieve the effect of distinguishing different fields.

[0145] In this embodiment, the OCR method can be used to recognize the text lines in the schematic diagram of the patient's expense list, as well as the coordinate positions of each single character in the original image and the recognition results of each field. Exemplarily, Figure 8 For Figure 7 the schematic diagram of the text detection results in the shown patient's expense list. As Figure 8 shown, take the text line of "Dressing (drawn silk gauze) 16 layers unknown block 2.1100 4 8.4400 8.4400" as an example for illustration.

[0146] Exemplarily, Figure 9 is the block diagram of the electronic device for image text recognition in the embodiment of the present application. As Figure 9 shown, in this embodiment, the electronic device includes: a semantic information encoding module, a character relative position encoding module, an information fusion module, a bidirectional LSTM module, and a CRF module. In this embodiment, the input of the electronic device is each single character in the text line and the coordinate position of the single character in the original image. The coordinate positions of each single character in the text line are as follows:

[0147]

[0148] In this embodiment, the semantic information encoding module extracts "word", that is, each single character, from the input, and learns the context semantics through a self-attention model, and finally outputs the semantic encoding vector corresponding to each single character.

[0149] The character relative position encoding module extracts "rect", that is, the coordinate information of each single character, from the input, and calculates the relative distances of the word coordinates of the upper left corner point, the center point, and the lower right corner point of adjacent single characters through the character relative position calculation module. Take the solution of the relative distance of the upper left corner point word coordinate of the character "material" as an example. The left relative distance is the left of "material" minus the left of "dressing": 493 - 461 = 32. Similarly, the top relative distance is 1242 - 1242 = 0. Similarly, the relative distances of the center point and the lower right corner point can be obtained. Concatenating the three distances can obtain a 6-dimensional vector. In this embodiment, in order to ensure that the relative position encoding information can be fused with the text semantic information encoded by the semantic information encoding module, a fully connected layer is used to map the 6-dimensional vector into the same dimension as the semantic encoding vector.

[0150] The information fusion module, that is, the semantic information and relative position fusion module, uses the semantic coding vector of each single word obtained by the semantic information coding module and the position coding vector obtained by the character relative position coding module, and fuses the two vectors by bitwise addition to obtain a synthetic coding vector.

[0151] The bidirectional LSTM module fully learns the input synthetic coding vector and determines the context information and category label of each word, and inputs it into the CRF module to obtain the output result corresponding to each word.

[0152] For example, Figure 10 for Figure 7 The diagram of the recognition result of the patient's expense list is shown in FIG. Figure 10 As shown in the figure, taking "Dressing (spun gauze) 16 layers unknown block 2.1100 4 8.4400" as an example, the output sequence is "B-Na I-Na I-Na I-Na I-Na I-Na I-Na E-Na B-Spec I-Spec I-Spec I-Spec E-Spec S-Un B-Num I-Num I-Num I-Num I-Num E-Num S-Num B-Num I-Num I-Num I-Num I-Num E-Num", where Na represents the name entity, Spec represents the specification entity, Un represents the unit entity, Num represents the number entity, B represents the start of the entity, I represents the middle of the entity, E represents the end of the entity, and S represents the single-word entity. Finally, by aggregating the output sequence, the final segmentation result of the field can be obtained as "Dressing (spun gauze) 16 layers unknown block 2.1100 4 8.4400".

[0153] The image text recognition method proposed in the embodiment of the present application encodes and fuses text and position information, and automatically determines the relationship between semantics and position. It can thus distinguish fields with larger pixel distances at the image level based on position coding, and can distinguish different fields with smaller distances but obvious semantic differences based on text semantic coding, which greatly facilitates the subsequent structuring of table text content.

[0154] The above describes the specific implementation of the image text recognition method mentioned in the embodiment of this application. The following is an embodiment of the device of this application, which can be used to execute the embodiment of the method of this application. For details not disclosed in the embodiment of the device of this application, please refer to the embodiment of the method of this application.

[0155] Figure 11This is a schematic diagram of the structure of the image text recognition device provided in the embodiment of the present application. The device can be integrated into or implemented by an electronic device, which can be a terminal device or a server. Figure 11 As shown, in this embodiment, the image text recognition device 110 may include: an acquisition module 1101 , a processing module 1102 and a determination module 1103 .

[0156] The acquisition module 1101 is configured to acquire an image to be processed, wherein the image to be processed contains at least one text line;

[0157] A processing module 1102 is configured to process each text line in the image to be processed to obtain a synthetic coding vector corresponding to each word in each text line, wherein the synthetic coding vector carries semantic information and position information;

[0158] The determination module 1103 is used to determine the text recognition result in the image to be processed according to the semantic information and position information carried by the synthetic coding vector corresponding to each single word in each text line.

[0159] In a possible design of an embodiment of the present application, the processing module 1102 is specifically used to analyze the semantics of each word in each text line in the image to be processed, obtain the semantic coding vector corresponding to each word in each text line, determine the coordinate information of each word in each text line in the image to be processed, encode and latitude-transform the coordinate information of each word, obtain the position coding vector of each word, the position coding vector has the same dimension as the semantic coding vector, and fuse the semantic coding vector and the position coding vector of each word to obtain a synthetic coding vector for each word.

[0160] In an embodiment of the present application, the processing module 1102 is configured to analyze the semantics of each word in each text line in the image to be processed to obtain a semantic encoding vector corresponding to each word in each text line, specifically:

[0161] The processing module 1102 is specifically configured to weightedly fuse the semantic information of all words in the text line to which each word in the image to be processed belongs, so as to obtain a semantic encoding vector of the word.

[0162] In an embodiment of the present application, the processing module 1102 is used to encode and latitude-transform the coordinate information of each word to obtain a position encoding vector of each word, specifically:

[0163] The processing module 1102 is specifically used to determine multiple pixel points representing each single word in the image to be processed, determine the position vector of each single word based on the coordinate information of the multiple pixel points corresponding to each single word, and perform dimension transformation on the position vector of each single word based on the dimension of the semantic coding vector corresponding to each single word to obtain a position coding vector with the same dimension as the semantic coding vector.

[0164] In another possible design of an embodiment of the present application, the determination module 1103 is specifically used to input the synthetic coding vector corresponding to each single word in each text line into a preset information analysis network to determine the context information of each single word. The preset information analysis network is used to analyze the context information of each single word, and determine the category label to which each single word belongs based on the context information of each single word. The context information of each single word in each text line and the category label to which each single word belongs are input into the discriminant model to obtain the text recognition result in the image to be processed.

[0165] In another possible design of an embodiment of the present application, the processing module 1102 is further used to perform optical character recognition on the image to be processed before processing each text line in the image to be processed to obtain a synthetic coding vector corresponding to each single word in each text line, and determine the position information of the text lines in the image to be processed and each single word in each text line.

[0166] In another possible design of an embodiment of the present application, the determination module 1103 is also used to determine the scope of each field in each text line based on the text recognition results in the image to be processed, each field includes multiple words that are mutually related in contextual semantic information, and the actual meaning of the text in the image to be processed is determined based on the contextual semantic information of all words in the same field in each text line.

[0167] The device provided in the embodiment of the present application can be used to perform Figures 1 to 10 The implementation principles and technical effects of the methods in the illustrated embodiments are similar and will not be described in detail here.

[0168] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by a processing element; or they can all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the determination module can be a separately established processing element, or it can be integrated into a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called by a processing element of the above device to perform the functions of the above determination module. The implementation of other modules is similar. In addition, these modules can all or partly be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0169] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code on a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)).

[0171] Furthermore, according to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0172] Figure 12 is a block diagram of an electronic device for implementing the image text recognition method of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0173] like Figure 12As shown, the electronic device includes: one or more processors 1201, a memory 1202, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 12 A processor 1201 is taken as an example.

[0174] Memory 1202 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor, causing the at least one processor to perform the image text recognition method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the image text recognition method provided in this application.

[0175] The memory 1202 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the image text recognition method in the embodiment of the present application (for example, the attached Figure 11 The processor 1101 executes the non-transient software programs, instructions, and modules stored in the memory 1102 to execute various functional applications and data processing of the server, thereby implementing the image text recognition method in the above method embodiment.

[0176] The memory 1202 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the electronic device for image text recognition, etc. In addition, the memory 1202 may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1202 may optionally include a memory remotely located relative to the processor 1201, and these remote memories may be connected to the electronic device for image text recognition via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The electronic device for image text recognition may further include: an input device 1203 and an output device 1204. The processor 1201, the memory 1202, the input device 1203 and the output device 1204 may be connected via a bus or other means. Figure 12 The bus connection is taken as an example.

[0178] The input device 1203 can receive input digital or character information and generate key signal input related to user settings and function control of the electronic device for image text recognition, such as input devices such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 1204 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0179] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0180] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0181] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0182] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0183] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0184] The present application also provides an image text recognition method, including:

[0185] Determine a synthetic encoding vector for each word in at least one text line of the image to be processed, wherein the synthetic encoding vector carries semantic information and position information;

[0186] The synthetic coding vector corresponding to each single word in each text line is processed to obtain the text recognition result in the image to be processed.

[0187] The specific implementation principle of this embodiment can be found in the above Figures 1 to 10 The description of the illustrated embodiments will not be repeated here.

[0188] According to the technical solution of the embodiment of the present application, an image to be processed is obtained, wherein the image to be processed contains at least one text line, and each text line in the image to be processed is processed to obtain a synthetic coding vector corresponding to each individual word in each text line. The synthetic coding vector carries semantic information and position information. Based on the semantic information and position information carried by the synthetic coding vector corresponding to each individual word in each text line, the text recognition result in the image to be processed is determined. This technical solution not only utilizes the semantic features of the text, but also integrates the position information of the text on the image. It can accurately distinguish adjacent fields in the image with a small pixel spacing, thereby improving the accuracy of text recognition in the image.

[0189] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0190] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A method for image character recognition, characterized in that: include: Acquire an image to be processed, wherein the image to be processed contains at least one text line; Processing each text line in the image to be processed to obtain a synthetic coding vector corresponding to each word in each text line, wherein the synthetic coding vector carries semantic information and position information, and is obtained by fusing the semantic coding vector and the position coding vector of each word based on the principle of same-position superposition, wherein the semantic coding vector of each word is obtained by weighted fusion of the semantic information of all words in the text line to which each word belongs, and the position coding vector of each word is obtained by encoding and dimensionally transforming the coordinate information of each word; The text recognition result in the image to be processed is determined based on the semantic information and position information carried by the synthetic coding vector corresponding to each single word in each text line.

2. The method according to claim 1, characterized in that The step of processing each text line in the image to be processed to obtain a synthetic coding vector corresponding to each word in each text line includes: Analyzing the semantics of each word in each text line in the image to be processed to obtain a semantic encoding vector corresponding to each word in each text line; Determining coordinate information of each word in each text line in the image to be processed; Encoding and dimensionally transforming the coordinate information of each word to obtain a position encoding vector of each word, wherein the position encoding vector has the same dimension as the semantic encoding vector; The semantic encoding vector and position encoding vector of each word are fused to obtain the synthetic encoding vector of each word.

3. The method according to claim 2, characterized in that The encoding and dimension transformation of the coordinate information of each word to obtain the position coding vector of each word includes: Determining a plurality of pixel points representing each word in the image to be processed; Determine the position vector of each word based on the coordinate information of multiple pixel points corresponding to each word; According to the dimension of the semantic coding vector corresponding to each single word, the position vector of each single word is transformed in dimension to obtain a position coding vector with the same dimension as the semantic coding vector.

4. The method according to any one of claims 1 to 3, characterized in that Determining the text recognition result in the image to be processed according to the semantic information and position information carried by the synthetic coding vector corresponding to each single word in each text line includes: Inputting the synthesized encoding vector corresponding to each word in each text line into a preset information analysis network to determine the context information of each word, wherein the preset information analysis network is used to analyze the context information of each word; Determine the category label of each word based on the context information of each word; The context information of each word in each text line and the category label to which each word belongs are input into the discriminant model to obtain the text recognition result in the image to be processed.

5. The method according to any one of claims 1 to 3, characterized in that Before processing each text line in the image to be processed to obtain a synthetic coding vector corresponding to each word in each text line, the method further includes: Optical character recognition is performed on the image to be processed to determine the text lines in the image to be processed and the position information of each word in each text line.

6. The method according to claim 1, characterized in that The method further comprises: Determining the range of each field in each text line according to the text recognition result in the image to be processed, each field including a plurality of words associated with each other in contextual semantic information; The actual meaning of the text in the image to be processed is determined according to the contextual semantic information of all words in the same field in each text line.

7. An image character recognition device, characterized in that: include: Acquisition module, processing module and determination module; The acquisition module is used to acquire an image to be processed, wherein the image to be processed contains at least one text line; The processing module is used to process each text line in the image to be processed to obtain a synthetic coding vector corresponding to each word in each text line, wherein the synthetic coding vector carries semantic information and position information, and is obtained by fusing the semantic coding vector and the position coding vector of each word based on the principle of superposition at the same position, wherein the semantic coding vector of each word is obtained by weighted fusion of the semantic information of all words in the text line to which each word belongs, and the position coding vector of each word is obtained by encoding and dimensionally transforming the coordinate information of each word; The determination module is used to determine the text recognition result in the image to be processed based on the semantic information and position information carried by the synthetic coding vector corresponding to each single word in each text line.

8. The device according to claim 7, characterized in that The processing module is specifically used to analyze the semantics of each word in each text line in the image to be processed, obtain the semantic coding vector corresponding to each word in each text line, determine the coordinate information of each word in each text line in the image to be processed, encode and dimensionally transform the coordinate information of each word to obtain the position coding vector of each word, the position coding vector has the same dimension as the semantic coding vector, and fuse the semantic coding vector and position coding vector of each word to obtain a synthetic coding vector for each word.

9. The device according to claim 8, characterized in that The processing module is used to encode and dimensionally transform the coordinate information of each word to obtain the position coding vector of each word, specifically: The processing module is specifically used to determine multiple pixel points representing each single word in the image to be processed, determine the position vector of each single word based on the coordinate information of the multiple pixel points corresponding to each single word, and perform dimension transformation on the position vector of each single word based on the dimension of the semantic coding vector corresponding to each single word to obtain a position coding vector with the same dimension as the semantic coding vector.

10. The device according to any one of claims 7 to 9, characterized in that: The determination module is specifically used to input the synthetic coding vector corresponding to each single word in each text line into a preset information analysis network to determine the context information of each single word. The preset information analysis network is used to analyze the context information of each single word, and determine the category label to which each single word belongs based on the context information of each single word. The context information of each single word in each text line and the category label to which each single word belongs are input into the discriminant model to obtain the text recognition result in the image to be processed.

11. The device according to any one of claims 7 to 9, characterized in that: The processing module is further used to perform optical character recognition on the image to be processed before processing each text line in the image to be processed to obtain the synthetic coding vector corresponding to each single word in each text line, and determine the position information of the text lines in the image to be processed and each single word in each text line.

12. The device according to claim 7, characterized in that The determination module is further configured to determine the scope of each field in each text line based on the text recognition results in the image to be processed, where each field includes multiple words whose contextual semantic information is mutually related, and to determine the actual meaning of the text in the image to be processed based on the contextual semantic information of all words in the same field in each text line.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

Images