A method and system for image and text recognition based on deep learning model

Abstract

The present application discloses a method and system for image and text recognition based on a deep learning model, and the method includes: performing image recognition on a test image to obtain a text box area; obtaining the height, width and length of the text box area; determining the sorting keyword of the text box area based on the height, width and length of the text box area; performing a first sorting on all text box areas according to the sorting keyword; preprocessing the text box areas after the first sorting; performing text recognition on the preprocessed text box areas based on a deep learning OCR model to obtain a recognition result of each preprocessed text box area; in the case of multiple recognition results, calculating the similarity between each recognition result and the expected text in a preset text library, and determining the recognition result by the similarity; merging all preprocessed text box areas to obtain a merged recognition result. The present application can effectively improve the accuracy of image and text recognition.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of optical character recognition, and in particular to a method and system for image and text recognition based on a deep learning model. Background Art

[0002] Deep learning technology is one of the most popular and cutting-edge technologies in the field of artificial intelligence in recent years. It is a machine learning algorithm based on neural networks and has been widely used in computer vision, natural language processing, speech recognition and other fields. It has powerful prediction, classification, clustering and other capabilities. With the rapid development of deep learning technology, the application of deep learning technology in image and text recognition is particularly prominent.

[0003] Traditional image and text recognition uses computer vision pattern recognition technology to automatically identify and extract characters from images or documents. However, traditional image and text recognition technology may cause text confusion during the recognition process. For example, during word splitting or merging, due to recognition errors or typesetting problems, traditional image and text recognition technology may split a word into multiple parts or merge multiple words into one. Secondly, there is also the phenomenon of line and paragraph confusion. Traditional image and text recognition technology may mistakenly identify the line and paragraph structure of the text, resulting in chaotic typesetting of the recognized text, which reduces the recognition accuracy.

[0004] Therefore, there is an urgent need for an image and text recognition method and system based on a deep learning model to improve the accuracy of image and text recognition. Summary of the invention

[0005] The embodiments of the present application provide a method and system for image and text recognition based on a deep learning model, which are used to improve the accuracy of image and text recognition.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions: In a first aspect, a method for image and text recognition based on a deep learning model is provided, the method comprising: In response to receiving the image to be tested, performing image recognition on the image to be tested to obtain at least one text box area; Get the height, width and length of each text box area; For any one of the text box areas, determining a sorting keyword for the text box area based on the height, width and length of the text box area; Performing a first sort on all the text box areas according to the sorting keyword to obtain the text box areas after the first sorting; Preprocessing the text box area after the first sorting to obtain a preprocessed text box area; Based on the deep learning OCR model, perform text recognition on the preprocessed text box area to obtain a recognition result of each preprocessed text box area; For any of the preprocessed text box areas, when there are multiple recognition results, calculate the similarity between each recognition result and the expected text in the preset text library, and take the expected text whose similarity is greater than a preset similarity threshold as the recognition result; All the preprocessed text box areas are merged to obtain a merged recognition result.

[0007] In a possible implementation manner of the first aspect, determining a sorting key for the text box area based on the height, width, and length of the text box area includes: Determine whether the length of the text box area is greater than a preset expected text box length; If the length of the text box area is greater than the preset expected text box length, calculating the product of the height of the text box and the preset expected text box length; Multiplying the product by a preset coefficient to obtain a first result; Using the absolute value of the difference between the first result and the width of the text box area as a sorting key for the text box area; If the length of the text box area is not greater than the preset expected text box length, the product of the width and the height of the text box area is used as the sorting key of the text box area.

[0008] In a possible implementation manner of the first aspect, merging all the preprocessed text box areas to obtain a merged recognition result includes: Mapping all of the text box areas into a two-dimensional coordinate system; According to the x-coordinates of all the text box areas, all the text box areas are sorted for the second time to obtain the text box areas after the second sorting, and the x-coordinate of each text box area is used as a second sorting key corresponding thereto; Construct an empty list and initialize the dictionary variable; For each of the text box areas, a merging step is performed in the order of the second sorting until all the text box areas complete the merging step, wherein, for each of the text box areas, the merging step is stored in the dictionary variable before the merging step is performed, and after the merging step is performed, the text box area is transferred to the empty list, and after at least one of the text box areas is transferred to the empty list, the empty list becomes a text box list; The merging step comprises: Traversing the text box list, determining whether the distance between any one of the text box areas in the text box list and the text box area in the dictionary variable is less than a preset distance; If it is less than the preset distance, determine that the text box area in the text box list and the text box area in the dictionary variable meet the merging condition, merge the text box area in the dictionary variable with the text box area in the text box list that meets the merging condition, and update the text box area information; If it is not less than the preset distance, the text box area in the dictionary variable is added to the text box list according to the second sorting keyword.

[0009] In a possible implementation manner of the first aspect, the method further includes: After traversing the text box list, determining whether the dictionary variable is empty; When the dictionary variable is empty, the merging step ends.

[0010] In a possible implementation manner of the first aspect, preprocessing the text box area after the first sorting to obtain the preprocessed text box area includes: Grayscale the text box area to obtain a grayscale text box image; De-noising the grayscale text box image by using a preset filtering technology to obtain a de-noised grayscale text box image; The denoised grayscale text box image is subjected to image enhancement processing to obtain a preprocessed text box area.

[0011] In a possible implementation manner of the first aspect, before performing text recognition on the preprocessed text box area by using the deep learning-based OCR model to obtain a recognition result of each preprocessed text box area, the method includes: Extracting edge features of the text box area; Determine the outline of the text box area through edge features; Using the outline, identifying a background image of the text box area; The background image and the text box area are segmented to obtain a text box area.

[0012] In a possible implementation manner of the first aspect, the deep learning-based OCR model is used to determine all text positions in the text box area and recognize text.

[0013] In a possible implementation manner of the first aspect, calculating the similarity between each recognition result and an expected text in a preset text library includes: For each of the recognition results, convert the recognition text into a text vector by using a pre-trained word vector model to obtain a first text vector; By using a pre-trained word vector model, any expected text in a preset text library is converted into a text vector to obtain a second text vector; The Euclidean distance between the first text vector and the second text vector is calculated, and the Euclidean distance is used as the similarity between the first text vector and the second text vector.

[0014] In a second aspect, the present application provides a machine-readable storage medium, the instructions of which are used to enable a machine to execute the above-mentioned image and text recognition method based on a deep learning model.

[0015] In a third aspect, the present application provides a graphic and text recognition system based on a deep learning model, comprising: a memory configured to store instructions; and The processor is configured to call the instructions from the memory and implement the above-mentioned image and text recognition method based on the deep learning model when executing the instructions.

[0016] Through the above technical solution, by performing image recognition on the image to be tested, obtaining at least one text box area, and sorting and preprocessing the text box area, the recognition error can be further reduced and the accuracy of the recognition result can be improved. When there are multiple recognition results, the similarity between each recognition result and the expected text in the preset text library is calculated, and the expected text with a similarity greater than the preset similarity threshold is used as the recognition result, which can further ensure the accuracy of the recognition result. All preprocessed text box areas are merged to obtain a merged recognition result, which can accurately recognize the text content in the image and integrate it into the merged recognition result, which is convenient for users to view and use, and improves the accuracy of the recognition result.

[0017] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a flow chart of a method for image and text recognition based on a deep learning model provided in an embodiment of the present application; Figure 2 A structural diagram of a deep learning OCR model text recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0020] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0021] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] Figure 1 The flowchart of a method for image and text recognition based on a deep learning model according to an embodiment of the present application is schematically shown. Figure 1 As shown, an embodiment of the present application provides a method for image and text recognition based on a deep learning model, which may include the following steps.

[0023] S110, in response to receiving the image to be tested, performing image recognition on the image to be tested to obtain at least one text box area; S120, obtaining the height, width and length of each text box area; S130, for any text box area, determining a sorting keyword for the text box area based on the height, width and length of the text box area; S140, performing a first sort on all text box areas according to the sorting keyword to obtain the text box areas after the first sorting; S150, preprocessing the text box area after the first sorting to obtain a preprocessed text box area; S160, performing text recognition on the preprocessed text box area based on the deep learning OCR model to obtain a recognition result of each preprocessed text box area; S170, for any preprocessed text box area, when there are multiple recognition results, calculating the similarity between each recognition result and the expected text in the preset text library, and taking the expected text with a similarity greater than a preset similarity threshold as the recognition result; S180: Merge all pre-processed text box areas to obtain a merged recognition result.

[0024] In response to receiving the image to be tested, image recognition is performed on the image to be tested to obtain at least one text box area. That is, when the image to be tested is received, image recognition can be performed on the image to be tested by an image recognition algorithm, such as a convolutional neural network, to obtain at least one text box area.

[0025] After obtaining at least one text box area, the height, width and length of each text box area are obtained. The boundary of the text box area can be identified by image processing algorithms such as edge detection. Based on the boundary information, the height, width and length of the text box area can be obtained.

[0026] Subsequently, for any text box area, the sorting key of the text box area is determined based on the height, width and length of the text box area. Specifically, first, it is determined whether the length of the text box area is greater than the preset expected text box length. In this embodiment, the preset expected text box length can be determined according to actual conditions; when the length of the text box area is greater than the preset expected text box length, the product of the height of the text box and the preset expected text box length is calculated, and the product is multiplied by a preset coefficient to obtain a first result. In this embodiment, the preset coefficient can be determined according to actual conditions. The absolute value of the difference between the first result and the width of the text box area is used as the sorting key of the text box area. When the length of the text box area is not greater than the preset expected text box length, the product of the width and height of the text box area is used as the sorting key of the text box area.

[0027] After obtaining the text box area after the first sort, all text box areas are sorted for the first time according to the sorting keyword to obtain the text box area after the first sort. That is to say, all text box areas can be sorted by the sorting keyword. After the sorting is completed, a text box area sorted by the sorting keyword is obtained.

[0028] The text box area after the first sorting is preprocessed. The preprocessing may include noise removal and normalization of the sorted text box area. Noise removal is to identify and remove non-text elements in the text box area, such as images, icons, lines, etc., which can be achieved through image processing techniques such as morphological operations and color segmentation. The text box area is normalized, such as scaling, rotating or cropping the text box area to meet the input requirements of the subsequent text recognition algorithm to ensure that they have consistent size and resolution. Subsequently, the preprocessed text box area is obtained.

[0029] Figure 2 A structural schematic diagram of text recognition of a deep learning OCR model provided for an embodiment of the present application, as shown in the figure, first, determine the text position in the image, and then, recognize the text characters in the text image. Based on the deep learning OCR model, text recognition is performed on the preprocessed text box area to obtain the recognition result of each preprocessed text box area. In this embodiment, the deep learning OCR model is an OCR model constructed using deep learning technology, such as convolutional neural network, recurrent neural network and other technologies, wherein the deep learning OCR model is used to determine all text positions in the text box area and recognize text. In other words, by performing text recognition on the preprocessed text box area through the deep learning OCR model, the recognition result of each preprocessed text box area can be obtained.

[0030] For any preprocessed text box area, when there are multiple recognition results, calculate the similarity between each recognition result and the expected text in the preset text library, and take the expected text with a similarity greater than the preset similarity threshold as the recognition result. In this embodiment, the expected text refers to the text stored in the preset text library, and the preset similarity threshold can be determined according to the actual situation. In the specific implementation process, the texts stored in the preset text library are all normalized texts, but because the recognition results output by the OCR model may be multiple, and among the multiple recognition results, there is only one normalized text, and the rest of the recognition results are non-normalized texts, the final recognition result can be determined by the preset text library. For example, if the recognition results are "market evaluation", "market evaluation side", and "city Yang evaluation", the normalized text can be determined as "market evaluation" through the preset text library. In summary, in this embodiment, if there are multiple recognition results, the deep learning OCR model recognizes the preprocessed text box area, which may generate multiple recognition results. The final recognition result can be determined by calculating the similarity between the text of each recognition result and the matched text in the preset text library, wherein the final recognition result refers to the expected text in the preset text library whose similarity is greater than the preset similarity threshold. The calculation of similarity can be achieved by the cosine similarity method, which evaluates the similarity of two vectors by calculating the cosine value of the angle between them.

[0031] All pre-processed text box areas are merged to obtain a merged recognition result. That is, the text box areas are merged through information such as the position, size and direction of the text box to form a continuous text box area, forming a new text box area, which may be a coherent text string.

[0032] Through the above technical solution, by performing image recognition on the image to be tested, obtaining at least one text box area, and sorting and preprocessing the text box area, the recognition error can be further reduced and the accuracy of the recognition result can be improved. When there are multiple recognition results, the similarity between each recognition result and the expected text in the preset text library is calculated, and the expected text with a similarity greater than the preset similarity threshold is used as the recognition result, which can further ensure the accuracy of the recognition result. All preprocessed text box areas are merged to obtain a merged recognition result, which can accurately recognize the text content in the image and integrate it into the merged recognition result, which is convenient for users to view and use, and improves the accuracy of the recognition result.

[0033] In one implementation of this embodiment, determining a sorting key of the text box area based on the height, width and length of the text box area includes: S210, determining whether the length of the text box area is greater than a preset expected text box length; S220: If the length of the text box area is greater than the preset expected text box length, calculate the product of the height of the text box and the preset expected text box length; S230, multiplying the product by a preset coefficient to obtain a first result; S240, using the absolute value of the difference between the first result and the width of the text box area as a sorting key for the text box area; S250: If the length of the text box area is not greater than the preset expected text box length, the product of the width and height of the text box area is used as a sorting key for the text box area.

[0034] Based on the height, width and length of the text box area, determine the sorting keyword of the text box area. Specifically, first, determine whether the length of the text box area is greater than the preset expected text box length. In this embodiment, the text box area generally refers to the part of the image that contains text content. The preset expected text box length can be determined according to actual conditions, that is, compare the detected length of the text box area with the preset expected text box length.

[0035] If the length of the text box area is greater than the preset expected text box length, the product of the height of the text box and the preset expected text box length is calculated. First, it is necessary to determine whether the length of the text box area is greater than the preset expected text box length. If the condition is met, that is, the length of the text box area is greater than the preset value, the next step of calculation is performed. The height of the text box is multiplied by the preset expected text box length, and then the product of the height of the text box and the preset expected text box length is multiplied by a preset coefficient. The preset coefficient multiplication can be determined according to the actual situation to obtain the first result.

[0036] After obtaining the first result, the absolute value of the difference between the first result and the width of the text box area is used as the sorting key of the text box area. That is to say, the difference between the first result and the width of the text box area is calculated. The difference can be a positive number or a negative number. The absolute value of the difference is used as the sorting key of the text box area. The sorting key is used to determine the position of the text box area in the sorting list.

[0037] When the length of the text box area is not greater than the preset expected text box length, the product of the width and height of the text box area is used as the sorting keyword of the text box area. Specifically, first, determine whether the width of the text box area is less than or equal to the preset expected width. If the condition is not met, that is, the width of the text box is greater than the preset expected text box length, the product of the width and height of the text box area is used as the sorting keyword of the text box area. The sorting keyword is used to determine the position of the text box area in the sorting list.

[0038] By determining the sorting keywords based on the height and width of the text box area, it is possible to flexibly adapt to text box areas of different sizes, optimize the text processing flow, enhance data visualization effects, and improve the accuracy of image and text recognition.

[0039] In one implementation of this embodiment, all pre-processed text box areas are merged to obtain a merged recognition result, including: S310, mapping all text box areas to a two-dimensional coordinate system; S320, performing a second sorting on all the text box areas according to the x-coordinates of all the text box areas to obtain the text box areas after the second sorting, and using the x-coordinate of each text box area as a second sorting key corresponding thereto; S330, construct an empty list and initialize the dictionary variable; S340, for each text box area, a merging step is performed in the order of the second sorting until all text box areas complete the merging step, wherein for each text box area, it is stored in a dictionary variable before the merging step is performed, and after the merging step is performed, the text box area is transferred to an empty list, and after at least one text box area is transferred to the empty list, the empty list becomes a text box list; The merging steps include: S350, traversing the text box list, determining whether the distance between any text box area in the text box list and the text box area in the dictionary variable is less than a preset distance; S360: If the distance is less than the preset distance, determine that the text box area in the text box list and the text box area in the dictionary variable meet the merging condition, merge the text box area in the dictionary variable with the text box area in the text box list that meets the merging condition, and update the text box area information; S370: If the distance is not less than the preset distance, the text box area in the dictionary variable is added to the text box list according to the second sorting keyword.

[0040] Map all text box areas to a two-dimensional coordinate system. First, determine a two-dimensional coordinate system and map all text box areas to the two-dimensional coordinate system, that is, convert each text box to the coordinates in the two-dimensional coordinate system. Then, sort all text box areas for a second time according to the x-coordinates of all text box areas to obtain the text box areas after the second sorting, and use the x-coordinate of each text box area as the corresponding second sorting key. Specifically, first, extract the x-coordinate of each text box area from the bounding box position information of each text box area, create a list containing all text box areas and their corresponding x-coordinates, and this list can be a tuple list, where each tuple contains a text box area and its x-coordinate. Use Python's built-in sorted function or the sort method of the list to sort all text box areas according to the x-coordinate to obtain the text box areas after the second sorting, and use the x-coordinate of each text box area as the corresponding second sorting key.

[0041] Then, construct an empty list and initialize the dictionary variable, that is, create an empty list, which can be called text_box_list, to store the processed text box area. Initialize a dictionary variable, called temp_dict, as a temporary storage to save the current text box area before the merging step.

[0042] Next, for each text box area, the merging step is performed in the order of the second sorting until all text box areas complete the merging step, wherein, for each text box area, it is stored in a dictionary variable before the merging step is performed, and after the merging step is performed, the text box area is transferred to an empty list. After at least one text box area is transferred to the empty list, the empty list becomes a text box list. Specifically, first, a loop is used to traverse each text box area in sorted_text_boxes. In each loop, the current text box area is used as a key, that is, a unique identifier is used as a key, and its value is the text box area itself, which is stored in temp_dict. This step is to retain the original state of the text box area before performing the merging operation. Secondly, the merging step is performed in the order of the second sorting until all text box areas have completed the merging step. After the merging step is completed, the processed text box areas are taken out of temp_dict and added to text_box_list. At this time, text_box_list begins to accumulate processed text box areas and continues to loop until all text box areas in sorted_text_boxes have passed the merging step and transferred to text_box_list. At this time, text_box_list is no longer an empty list, but contains all text box areas processed by the merging step, which is called the text box list.

[0043] The merging step specifically traverses the text box list to determine whether the distance between any text box area in the text box list and the text box area in the dictionary variable is less than a preset distance, that is, each text box area in the text box list is checked one by one to determine whether the distance between them and any text box area in the dictionary variable is less than a preset distance.

[0044] If it is less than the preset distance, determine that the text box area in the text box list and the text box area in the dictionary variable meet the merging condition, merge the text box area in the dictionary variable with the text box area in the text box list that meets the merging condition, and update the text box area information, that is, if the distance is less than the preset distance, it can be considered that they meet the merging condition. Merging the two text box areas usually means merging their position and size information into a new text box area, which should be able to contain the original two areas, and update the text box area information.

[0045] If it is not less than the preset distance, the text box area in the dictionary variable is added to the text box list according to the second sorting keyword. That is to say, if the distance between the text box area in the dictionary variable and any area in the text box list is not less than the preset distance, then the text box areas in these dictionaries can be added to the text box list according to a second sorting keyword.

[0046] By merging each text box area, the clarity and readability of the recognition results can be improved, the text order can be maintained, and the recognition accuracy can be improved.

[0047] In one implementation of this embodiment, the method further includes: S410, after traversing the text box list, determining whether the dictionary variable is empty; S420. When the dictionary variable is empty, end the merging step.

[0048] After traversing the text box list, determine whether the dictionary variable is empty, that is, after executing all the merging steps, that is, traversing the text box list and comparing and merging with the text box areas in the dictionary variable, it is necessary to check whether the dictionary variable still contains text box areas; in the case that the dictionary variable is empty, end the merging step. If the dictionary variable is empty, it means that all text box areas have been processed, then the merging step can be safely ended, which means that there are no more text box areas to be processed and the merging process has been completed.

[0049] By judging whether the dictionary variable is empty, and ending the merging step when the dictionary variable is empty, it can be ensured that all text box areas have been properly processed, ensuring the integrity of the merge, avoiding redundant processing, and helping to improve the merge efficiency.

[0050] In one implementation of this embodiment, the text box area after the first sorting is preprocessed to obtain the preprocessed text box area, including: S510, graying the text box area to obtain a gray text box image; S520, performing denoising on the grayscale text box image by using a preset filtering technology to obtain a denoised grayscale text box image; S530 , performing image enhancement processing on the denoised grayscale text frame image to obtain a pre-processed text frame area.

[0051] First, the text box area is grayed to obtain a gray text box image. Graying is an image preprocessing technology that converts a color image into a gray image. The RGB value of each pixel in the color image is converted into a gray value. The conversion formula can be a weighted average method. After graying, the text box area that originally contained color information will be converted into an image containing only gray information, and a gray text box image is obtained.

[0052] Subsequently, the grayscale text box image is denoised by a preset filtering technology to obtain a denoised grayscale text box image. In this embodiment, the preset filtering technology can be a Gaussian filtering technology. Gaussian filtering technology is a smoothing filter suitable for removing Gaussian noise, and the degree of smoothing can be adjusted as needed; the grayscale text box image is denoised by a Gaussian filtering technology, and the denoised grayscale text box image is output.

[0053] The denoised grayscale text box image is subjected to image enhancement processing to obtain the preprocessed text box area. The image enhancement technology can be used to select a suitable image enhancement technology according to specific application requirements and image characteristics, such as histogram equalization. Histogram equalization can improve the contrast of the image, make its distribution more uniform, enhance the image picture, and obtain the preprocessed text box area.

[0054] By preprocessing the text box area after the first sorting, the text recognition accuracy can be improved, the processing time can be reduced, the image stability can be enhanced, and the image stability can be improved, making the subsequent image analysis tasks more reliable.

[0055] In one implementation of this embodiment, before performing text recognition on the preprocessed text box area through the deep learning-based OCR model to obtain the recognition result of each preprocessed text box area, the method includes: S610, extracting edge features of the text box area; S620, determining the outline of the text box area through edge features; S630, identifying the background image of the text box area through the outline; S640: Segment the background image and the text box area to obtain the text box area.

[0056] To extract the edge features of the text box area, you can use edge detection algorithms, such as Canny edge detection. Canny edge detection is an edge detection method widely used in computer vision and image processing. Edge detection is performed on the text box area to identify the pixel points on the edge of the text box and obtain the edge features of the text box area.

[0057] The outline of the text box area is determined through edge features. That is, based on the extracted edge features, a contour tracing algorithm, such as Hough transform, contour tree, etc., is used to connect edge pixels to form a complete text box outline and determine the outline of the text box area.

[0058] Subsequently, the background image of the text box area is identified through the outline. That is, based on the determined text box outline, the pixels outside the outline are regarded as part of the background image, the boundary between the text box area and the background image is identified, and the background image of the text box area is determined.

[0059] The background image and the text box area are segmented to obtain the text box area. Image segmentation techniques, such as threshold segmentation, region growing, graph cutting algorithm, etc., can be used to separate the background image and the text box area according to the text box outline to obtain the text box area.

[0060] By segmenting the background image from the text box area, the text box area is obtained, which not only improves the accuracy and efficiency of OCR model recognition, but also enhances the robustness of the model and user experience.

[0061] In one implementation of this embodiment, calculating the similarity between each recognition result and the expected text in the preset text library includes: S710: For each recognition result, convert the recognized text into a text vector by using a pre-trained word vector model to obtain a first text vector; S720, converting any expected text in a preset text library into a text vector by using a pre-trained word vector model to obtain a second text vector; S730: Calculate the Euclidean distance between the first text vector and the second text vector, and use the Euclidean distance as the similarity between the first text vector and the second text vector.

[0062] For each recognition result, the recognized text is converted into a text vector through a pre-trained word vector model to obtain the first text vector. First, the pre-processed text is converted into a text vector using a pre-trained word vector model. Each word or character is mapped to its corresponding vector representation by looking up the vocabulary in the model. If the text contains words or characters outside the model vocabulary, special strategies may need to be used, such as using unknown word vectors or fine-tuning word embeddings.

[0063] After conversion by the word vector model, each recognition result is converted into a vector representation of a fixed dimension, namely the first text vector.

[0064] Similarly, through the pre-trained word vector model, any expected text in the preset text library is converted into a text vector to obtain a second text vector. Using the pre-trained word vector model, the pre-processed expected text is converted into a text vector. By searching the vocabulary in the model, each word or character is mapped to its corresponding vector representation. If the expected text contains words or characters outside the model vocabulary, special strategies can be used to handle it, such as using unknown word vectors and fine-tuning word embeddings. After conversion by the word vector model, each expected text is converted into a vector representation of a fixed dimension, namely the second text vector.

[0065] Then, the Euclidean distance between the first text vector and the second text vector is calculated, and the Euclidean distance is used as the similarity between the first text vector and the second text vector. The Euclidean distance measures the straight-line distance between two vectors in multidimensional space. The closer the distance, the more similar the two vectors are. In other words, the similarity between the first text vector and the second text vector is calculated using the Euclidean distance formula, which is as follows:

[0066] in, represents the Euclidean distance, and express and The i-th component of .

[0067] By calculating the similarity between each recognition result and the expected text in the preset text library, the flexibility and accuracy of image and text recognition can be improved, the robustness of image and text recognition can be enhanced, and the efficiency and accuracy of image and text recognition can be guaranteed.

[0068] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned image and text recognition method based on the deep learning model.

[0069] The present application provides an image and text recognition system based on a deep learning model, comprising: a memory configured to store instructions; and The processor is configured to call instructions from the memory and implement the above-mentioned image and text recognition method based on the deep learning model when executing the instructions.

[0070] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0075] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0076] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0077] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0078] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for image and text recognition based on a deep learning model, characterized in that: include: In response to receiving the image to be tested, performing image recognition on the image to be tested to obtain at least one text box area; Get the height, width and length of each text box area; For any one of the text box areas, determining a sorting keyword for the text box area based on the height, width and length of the text box area; Performing a first sort on all the text box areas according to the sorting keyword to obtain the text box areas after the first sorting; Preprocessing the text box area after the first sorting to obtain a preprocessed text box area; Based on the deep learning OCR model, perform text recognition on the preprocessed text box area to obtain a recognition result of each preprocessed text box area; For any of the preprocessed text box areas, when there are multiple recognition results, calculate the similarity between each recognition result and the expected text in the preset text library, and take the expected text whose similarity is greater than a preset similarity threshold as the recognition result; All the preprocessed text box areas are merged to obtain a merged recognition result.

2. The method according to claim 1, characterized in that The step of determining a sorting keyword for the text box area based on the height, width and length of the text box area includes: Determine whether the length of the text box area is greater than a preset expected text box length; If the length of the text box area is greater than the preset expected text box length, calculating the product of the height of the text box and the preset expected text box length; Multiplying the product by a preset coefficient to obtain a first result; Using the absolute value of the difference between the first result and the width of the text box area as a sorting key for the text box area; If the length of the text box area is not greater than the preset expected text box length, the product of the width and the height of the text box area is used as the sorting key of the text box area.

3. The method according to claim 1, characterized in that The merging of all the pre-processed text box regions to obtain a merged recognition result includes: Mapping all of the text box areas into a two-dimensional coordinate system; According to the x-coordinates of all the text box areas, all the text box areas are sorted for the second time to obtain the text box areas after the second sorting, and the x-coordinate of each text box area is used as a second sorting key corresponding thereto; Construct an empty list and initialize the dictionary variable; For each of the text box areas, a merging step is performed in the order of the second sorting until all the text box areas complete the merging step, wherein, for each of the text box areas, the merging step is stored in the dictionary variable before the merging step is performed, and after the merging step is performed, the text box area is transferred to the empty list, and after at least one of the text box areas is transferred to the empty list, the empty list becomes a text box list; The merging step comprises: Traversing the text box list, determining whether the distance between any one of the text box areas in the text box list and the text box area in the dictionary variable is less than a preset distance; If it is less than the preset distance, determine that the text box area in the text box list and the text box area in the dictionary variable meet the merging condition, merge the text box area in the dictionary variable with the text box area in the text box list that meets the merging condition, and update the text box area information; If it is not less than the preset distance, the text box area in the dictionary variable is added to the text box list according to the second sorting keyword.

4. The method according to claim 3, characterized in that The method further comprises: After traversing the text box list, determining whether the dictionary variable is empty; When the dictionary variable is empty, the merging step ends.

5. The method according to claim 1, characterized in that The preprocessing of the text box area after the first sorting to obtain the preprocessed text box area includes: Grayscale the text box area to obtain a grayscale text box image; De-noising the grayscale text box image by using a preset filtering technology to obtain a de-noised grayscale text box image; The denoised grayscale text box image is subjected to image enhancement processing to obtain a preprocessed text box area.

6. The method according to claim 1, characterized in that Before performing text recognition on the preprocessed text box area by using the deep learning-based OCR model to obtain a recognition result of each preprocessed text box area, the method includes: Extracting edge features of the text box area; Determine the outline of the text box area through edge features; Using the outline, identifying a background image of the text box area; The background image and the text box area are segmented to obtain a text box area.

7. The method according to claim 1, characterized in that The deep learning-based OCR model is used to determine all text positions in the text box area and recognize text.

8. The method according to claim 1, characterized in that The calculating the similarity between each recognition result and the expected text in the preset text library includes: For each of the recognition results, convert the recognition text into a text vector by using a pre-trained word vector model to obtain a first text vector; By using a pre-trained word vector model, any expected text in a preset text library is converted into a text vector to obtain a second text vector; The Euclidean distance between the first text vector and the second text vector is calculated, and the Euclidean distance is used as the similarity between the first text vector and the second text vector.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for enabling a machine to execute the image and text recognition method based on a deep learning model according to any one of claims 1 to 8.

10. A graphic and text recognition system based on a deep learning model, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and to implement the image and text recognition method based on a deep learning model according to any one of claims 1 to 8 when executing the instructions.

Images