Text recognition method, device and storage medium

Abstract

The application relates to a character recognition method, a device and a storage medium. The character recognition method is applied to a terminal device, and the method comprises the following steps: displaying first content, wherein the first content is an image containing characters; receiving a character recognition operation performed by a user on the first content; determining an image evaluation level corresponding to the first content in response to the character recognition operation; and recognizing the first content by using a character recognition algorithm corresponding to the image evaluation level to obtain a character recognition result. According to the embodiment of the application, the character recognition algorithm corresponding to the image evaluation level is used to recognize the characters in the first content, so that the characters in the first content can be more accurately recognized, and unnecessary calculation amount is reduced.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a character recognition method, device, and storage medium. Background Technology

[0002] With the widespread use of terminal devices, images are increasingly used as a medium for information dissemination, leading to a growing demand for text recognition within images. Currently, optical character recognition (OCR) technology is commonly used to identify text in images. OCR technology utilizes text recognition algorithms to convert text in images (including typed text, printed text, and / or handwritten text) into computer text. As the content contained in images becomes increasingly complex and diverse, accurately recognizing text within images has become a crucial problem that urgently needs to be solved. Summary of the Invention

[0003] In view of this, embodiments of this application propose a character recognition method, device, and storage medium.

[0004] In a first aspect, embodiments of this application provide a text recognition method applied to a terminal device. The method includes: displaying first content, wherein the first content is an image containing text; receiving a text recognition operation performed by a user on the first content; in response to the text recognition operation, determining an image evaluation level corresponding to the first content; and recognizing the first content using a text recognition algorithm corresponding to the image evaluation level to obtain a text recognition result.

[0005] The text recognition method of this application can, after receiving a text recognition operation for a first content, first determine the image evaluation level of the first content, and then use the text recognition algorithm corresponding to the image evaluation level to perform text recognition on the first content, thereby more accurately recognizing the text in the first content and reducing unnecessary computation.

[0006] According to the first aspect, in a first possible implementation of the first aspect, the image evaluation level includes a first image evaluation level and a second image evaluation level. The step of using the text recognition algorithm corresponding to the image evaluation level to recognize the first content and obtain a text recognition result includes: when the image evaluation level is the first image evaluation level, using the first text recognition algorithm to recognize the first content and obtain a first text recognition result; or, when the image evaluation level is the second image evaluation level, using the second text recognition algorithm to recognize the first content and obtain a second text recognition result, and determining the second text recognition result as the final output text recognition result; wherein, the image with the first image evaluation level includes a screenshot image, the screenshot image includes regularly arranged and regularly sized text, and the image with the second image evaluation level includes a photograph image, the photograph image includes irregularly arranged and irregularly sized text.

[0007] In this embodiment, the text recognition method employs different text recognition algorithms based on the image evaluation level of the first content, enabling more accurate text recognition within the first content and reducing unnecessary computation. Furthermore, the method can also determine the image evaluation level based on the ease with which text can be recognized. That is, a screenshot image containing regularly arranged and regularly sized text is designated as the first image evaluation level, and the first text recognition algorithm is used to recognize the text. Conversely, a photographed image containing irregularly arranged and irregularly sized text is designated as the second image evaluation level, and the second text recognition algorithm is used to recognize the text. By employing these methods, the text recognition method can quickly determine the corresponding text recognition algorithm when performing text recognition on screenshots or photographed images, improving response and processing speed.

[0008] According to the first aspect, in a second possible implementation of the first aspect, the algorithmic complexity of the first character recognition algorithm is lower than that of the second character recognition algorithm, or the recognition speed of the first character recognition algorithm is faster than that of the second character recognition algorithm.

[0009] In this embodiment, the method can employ text recognition algorithms of varying complexity based on the image's evaluation level. If the image evaluation level is high (i.e., the first image evaluation level), a text recognition algorithm with low complexity or fast recognition speed is used, thus enabling faster text recognition while maintaining accuracy. If the image evaluation level is low (i.e., the second image evaluation level), a text recognition algorithm with high complexity or slow recognition speed is used, ensuring text recognition or a high text recognition rate.

[0010] According to the first aspect, in a second or third possible implementation of the first aspect, after recognizing the first content using a first character recognition algorithm to obtain a first character recognition result when the image evaluation level is the first image evaluation level, the method further includes: determining whether the first character recognition result meets a preset requirement; if the first character recognition result meets the preset requirement, determining the first character recognition result as the final output character recognition result; if the first character recognition result does not meet the preset requirement, recognizing the first content using a second character recognition algorithm to obtain a second character recognition result, and determining the second character recognition result as the final output character recognition result.

[0011] In this embodiment, since the algorithm complexity of the first character recognition algorithm is low, the character recognition effect of the first character recognition result may not be as expected. For example, no characters may be recognized or only a small number of characters may be recognized. Therefore, the character recognition effect of the first character recognition result can be judged, that is, whether the first character recognition result meets the preset requirements. If the preset requirements are met, the first character recognition result can be used as the final output character recognition result. If the preset requirements are not met, the second character recognition result recognized by the second character recognition algorithm with higher complexity can be used as the final output character recognition result, which increases the judgment of the effect of the character recognition result.

[0012] According to the first aspect, in the fourth possible implementation of the first aspect, the preset requirement includes that the intersection-union ratio of the text recognition region corresponding to the first text recognition result and the region of the first content is greater than a first preset threshold, and / or that the text integrity of the text recognition region corresponding to the first text recognition result and the first content region to which the first content belongs is greater than a second preset threshold.

[0013] In this embodiment, during the process of judging the text recognition effect of the first text recognition result, the method can quantify the text recognition effect, that is, use the intersection-union ratio (IUGR) to judge the first text recognition result to determine whether the text has been recognized. If the text has been recognized, the method can further determine the text recognition effect of the first text recognition result using text completeness. If the text recognition effect is not as expected (i.e., below a preset threshold), then the second text recognition result can be used. In summary, the method can use measurement indicators (including text completeness and / or IUGR) to quantify the text recognition effect, making the judgment more reliable.

[0014] According to the first aspect or the first to fifth possible implementations of the first aspect, determining the image evaluation level corresponding to the first content includes: inputting the first content into an image evaluation level determination model to obtain the image evaluation level corresponding to the first content, wherein the image evaluation level determination model is a model trained based on training data, the training data includes multiple training images and the image evaluation level corresponding to each training image, and the image evaluation level determination model includes a neural network structure.

[0015] In this embodiment, the method may employ an image rating level determination model to judge the first content. This image rating level determination model may employ a neural network structure and may be trained using training data, thereby more accurately determining the image rating level of the first content.

[0016] According to the first aspect or the sixth possible implementation of the first aspect, the image evaluation level determination model is a multi-task learning model, and the image evaluation level determination model is further used to obtain the category of the first content and the region information corresponding to the category.

[0017] In this embodiment, the method can employ a multi-task learning model as the image evaluation level determination model. This model can determine the image evaluation level of the first content while also acquiring the category of the first content and the corresponding region information. Thus, a single model can complete multiple tasks, meeting new demands generated within the terminal device and reducing the burden on the terminal device.

[0018] According to the first aspect or the first to seventh possible implementations of the first aspect, before displaying the first content, the method further includes: in response to the user's content selection operation on the first image, determining the first content, wherein the first content is part or all of the content in the first image.

[0019] In this embodiment, the user can select the first content to be recognized from the first image, thereby meeting the user's needs.

[0020] Secondly, embodiments of this application provide a terminal device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions using one or more of the character recognition methods described in the first aspect or various possible implementations of the first aspect.

[0021] The terminal device in the embodiments of this application can, after receiving a text recognition operation for the first content, first determine the image evaluation level of the first content, and then use the text recognition algorithm corresponding to the image evaluation level to perform text recognition on the first content, thereby more accurately recognizing the text in the first content and reducing unnecessary computation.

[0022] Thirdly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement one or more of the character recognition methods described in the first aspect or various possible implementations of the first aspect.

[0023] The embodiments of this application can, upon receiving a text recognition operation for a first content, first determine the image evaluation level of the first content, and then use a text recognition algorithm corresponding to the image evaluation level to perform text recognition on the first content, thereby more accurately recognizing the text within the first content and reducing unnecessary computation.

[0024] Fourthly, embodiments of this application provide a computer program product, the computer program product including instructions for instructing a processor to execute one or more of the character recognition methods described in the first aspect or various possible implementations of the first aspect.

[0025] The embodiments of this application can, upon receiving a text recognition operation for a first content, first determine the image evaluation level of the first content, and then use a text recognition algorithm corresponding to the image evaluation level to perform text recognition on the first content, thereby more accurately recognizing the text within the first content and reducing unnecessary computation.

[0026] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description

[0027] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0028] Figure 1 This illustration shows a scenario of performing text recognition using a terminal device according to an embodiment of this application;

[0029] Figure 2 The diagram illustrates the performance of character recognition according to an embodiment of this application;

[0030] Figure 3 This diagram illustrates a flowchart of the steps of a character recognition method according to an embodiment of this application.

[0031] Figure 4 A diagram illustrating an image evaluation level according to an embodiment of this application is shown;

[0032] Figure 5 A diagram illustrating the determination of an image evaluation level according to an embodiment of this application is shown.

[0033] Figure 6 A diagram illustrating a multi-task learning model according to an embodiment of this application is shown;

[0034] Figure 7 This diagram illustrates a method for determining character recognition results according to an embodiment of the present application.

[0035] Figure 8 A diagram illustrating text box information according to an embodiment of this application;

[0036] Figure 9 This diagram illustrates a flowchart of the steps of a character recognition method according to an embodiment of this application.

[0037] Figure 10 This diagram illustrates the structure of a terminal device according to an embodiment of the present application.

[0038] Figure 11 A software structure block diagram of a terminal device according to an embodiment of this application is shown. Detailed Implementation

[0039] The various embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0040] In this application embodiment, " / " can indicate that the related objects are in an "or" relationship. For example, A / B can represent A or B. "And / or" can be used to describe three relationships between related objects. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. To facilitate the description of the technical solutions in this application embodiment, the terms "first" and "second" can be used to distinguish technical features with the same or similar functions. These terms do not limit the quantity or execution order, and they are not necessarily different. In this application embodiment, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" should not be construed as being better or more advantageous than other embodiments or design solutions. The use of "exemplary" or "for example" is intended to present related concepts in a specific manner for ease of understanding.

[0041] Furthermore, to better illustrate this application, specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0042] Before explaining the embodiments of this application, the application scenarios of the embodiments of this application will be described first.

[0043] The terminal devices described in this application may include mobile phones, tablets, wearable devices (e.g., watches, bracelets, smart helmets, etc.), in-vehicle devices, smart home devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiments in this application use mobile phones as examples; it is understood that these embodiments can also be applied to other types of terminal devices.

[0044] The terminal device includes a screen, which may be a touchscreen. The screen may include, but is not limited to, display devices such as monitors, and may also include other devices for displaying images, such as mirroring devices or extended screens. In this application, the screen may display content to be processed, or it may display text recognized from that content.

[0045] The application (app) involved in this application, which can be simply referred to as an application, is a software program capable of performing one or more specific functions. Typically, multiple applications can be installed on a terminal device, such as sports applications, music applications, dictionary applications, etc. It is understood that an application can be an application pre-installed on the terminal device at the factory, or an application downloaded by the user from the network, obtained from other electronic devices, or a third-party application during the use of the terminal device. Third-party applications typically refer to applications developed by developers other than the terminal device manufacturer. For example, for an H-brand mobile phone manufactured by H company, a third-party application could be an application developed by developers other than H company.

[0046] As an example, the text recognition method provided in this application embodiment can be provided to users in the form of an application, interface, or system function. For example... Figure 1 As shown in (a), the terminal device is equipped with an application 101 capable of executing the text recognition method provided in this application embodiment. As another example, existing applications on the terminal device can call the text recognition interface corresponding to the text recognition method provided in this application embodiment. Taking an instant messaging application as an example, the terminal device can call the text recognition interface of the instant messaging application to perform text recognition on content of interest to the user. For example, the user can call the text recognition interface by long-pressing or pressing with two fingers, and then use the text recognition interface to perform text recognition on the currently displayed content of the application.

[0047] In implementation, terminal equipment, such as Figure 1 After launching application 101 as shown in (a), the main interface corresponding to application 101 can be displayed on the screen. On this main interface, the user can select a first image to be processed by clicking or triggering relevant controls (e.g., an image selection control). The first image can be an image stored in the terminal device, an image captured by the image acquisition device of the terminal device, or an image downloaded or received by the terminal device through a network.

[0048] It should be noted that the text recognition method provided in this application targets images. That is, the technical solution of this application is aimed at recognizing text from images. The images that can be processed in the embodiments of this application include images captured by screenshots, images captured by a camera, or images acquired from external sources, such as images downloaded or received by a user via a network using a terminal device. After the user selects the first image to be processed, the terminal device can... Figure 1 As shown in (b), the first image is displayed on the first user interface 102. The first image can be an image composed of content displayed on the screen of the terminal device by the first user interface 102, such as... Figure 1 As shown in (b), the first image may include multiple contents such as a title, picture, and article.

[0049] When the first image is displayed on the first user interface 102, the user can select the first content that the user wishes to be recognized, which may also be referred to as the region of interest (ROI). The terminal device can then execute the text recognition method provided in this application embodiment on the first content to recognize the text in the first content.

[0050] In implementation, a user can select first content from a first image through a content selection operation. The content selection operation refers to determining the first content using a user's body part (e.g., a finger) or input device (e.g., a stylus, keyboard, mouse, etc.). As an example, the terminal device can use a body part or input device to obtain the start and end positions of the first content, and then use the start and end positions as vertices of a polygon (e.g., a rectangle) to define a polygonal region as the first content.

[0051] As another example, such as Figure 1 As shown in (c), after the body part or input device inputs the starting position 33 in the first image, it continues to input, and the continuously input path 44 has formed a regular or irregular region 103. The terminal device can determine this regular or irregular region 103 as the first content.

[0052] As another example, the terminal device can directly perform text recognition on the first image after acquiring it. The terminal device can then determine the content selection area based on the content selection operation and display the text recognition result corresponding to the content selection area.

[0053] Subsequently, the terminal device can receive text recognition operations from the user on the first content. The text recognition operation can be a single action or a combination of multiple actions. For example, the user can use a body part or input device to click on the control corresponding to the text recognition function or service to perform the text recognition operation. As another example, the user can perform text recognition operations via voice commands. For instance, after the user says the voice command "Please recognize the text in the image," the terminal device responds to the command, performs voice recognition and intent understanding, and then executes the text recognition operation. Furthermore, the user can also perform text recognition operations through combined operations (e.g., long press, double-tap, triple-tap).

[0054] Furthermore, in practice, users can simultaneously perform content selection and text recognition operations through specific actions. For example... Figure 1As shown in (b), a user can select a first content by pressing with two fingers and then perform text recognition on that content. The two-finger pressing operation instructs the user to simultaneously press two points on the screen with two fingers for a preset time (e.g., two seconds). In implementation, the terminal device can determine the content corresponding to the area between the two fingers as the first content and perform text recognition on that content. To more accurately determine the first content, the pressing positions 11 and 22 of the user's two fingers can be displayed on the screen as a handle, which can be a graphical interface element. This allows the user to determine the first content by adjusting the positions of pressing positions 11 and 22.

[0055] Furthermore, in practice, the application may also provide related services based on the identified content. For example... Figure 1 As shown in (d), the application can also segment the recognized text and display the segmented characters, words, or punctuation marks as a mask 104 on the first user interface 102 to facilitate users in copying, searching, translating, or sharing one or more characters, words, or punctuation marks. The aforementioned mask refers to a single, complete layer superimposed on the current display interface, for example, in the form of a pop-up window. In implementation, the mask, or a portion thereof, can be displayed on the first user interface 102 in a transparent, semi-transparent, or opaque manner. Furthermore, the terminal device can also overlay the text recognition results on the first content or the first user interface 102 as a mask. The segmentation process can be performed by the server after the terminal uploads the recognized text, or it can be performed locally on the terminal.

[0056] In text recognition, the recognition rate varies depending on the complexity of the image. Different text recognition algorithms with varying complexity can be used depending on the image's complexity. In short, simpler images can use lower-complexity algorithms, i.e., simple OCR. More complex images require higher-complexity algorithms, i.e., complex OCR. Simple OCR algorithms have lower complexity than complex OCR algorithms; therefore, simple OCR processes faster, but its accuracy is generally lower. In this application, the specific text recognition algorithm used is not fixed but determined through comparisons between multiple algorithms. When there are two or more text recognition algorithms, the algorithm with low complexity, low text recognition accuracy, and fast processing speed can be called simple OCR, while the algorithm with high complexity, high text recognition accuracy, and fast processing speed can be called complex OCR.

[0057] When the image is relatively simple, simple OCR can be used to recognize the text in the image. This simple OCR can include traditional image processing algorithms. For example, a traditional image processing algorithm can determine the shape of the text in the image by detecting dark and light patterns, and then use character recognition methods to convert the shape into text, or in other words, match the shape to the corresponding text. Optionally, the matching can be achieved through feature extraction. This method is fast and has high accuracy for simple images, but performs poorly for complex images. The aforementioned simple image refers to an image where the text is arranged regularly and is of regular size; for example, an image where the text is arranged regularly in straight lines and is of the same size. Therefore, this text recognition method is generally suitable for recognizing text in screenshots, interface content, etc. The aforementioned complex image refers to an image containing text that is irregularly arranged and of irregular size; for example, an image where the text is arranged irregularly in a curved form and is of different sizes, such as a photograph taken by a terminal device through an image acquisition device.

[0058] In implementation, a screenshot / capture image can refer to a snapshot of the content displayed on the screen of a terminal device, creating an image with the same dimensions and content as the screen. For example, a user can take a screenshot by simultaneously pressing the power button and the home button briefly, or by simultaneously pressing the volume up / down buttons and the power button briefly. Alternatively, a screenshot can be taken by tapping the screen twice with a knuckle. Screenshot images typically include the content of application interfaces, system interfaces, etc., which usually contain regularly arranged text. In one possible implementation, the user can activate a text recognition service by performing a text recognition operation to recognize the text in the interface content.

[0059] In one possible implementation, after the user performs a snapshot operation, a first control indicating the completion of the screenshot, a preview image of the screenshot, and / or a second control for performing text recognition on the screenshot pops up on the terminal device's display interface; after the user clicks the second control, the text recognition result corresponding to the screenshot content is displayed on the terminal device's display interface; or, after the user clicks the first control or the preview image of the screenshot, the terminal device displays a screenshot preview interface, which includes a third control for performing text recognition on the screenshot, and then, after the user clicks the third control, the text recognition result corresponding to the screenshot content is displayed on the terminal device's display interface.

[0060] For complex images, especially photographs taken in natural environments, which may contain curved arrays, irregular surfaces, slope distributions, wrinkles, distortions, or incompleteness, the recognition results of simple OCR may not be ideal. Therefore, complex OCR can be used to perform text recognition processing on complex images. As mentioned above, the algorithm complexity of complex OCR is higher than that of simple OCR, and the processing speed of complex OCR is slower than that of simple OCR. Consequently, the text recognition accuracy of complex OCR is generally higher than that of simple OCR. In the following description, artificial intelligence (AI) will be used as an example to describe complex OCR. It should be understood that the complex OCR described in the embodiments of this application is not limited to OCR implemented with an AI model.

[0061] AI technology can include model training and model inference (or prediction) processes. To obtain data for training the model (training data), AI technology can also include a data acquisition (or collection) process. The AI ​​technology involved in this application embodiment can be implemented using machine learning (ML) models, deep learning (DL) models, etc. AI technology can train models using training data in a data-driven manner, endowing machines with specific capabilities, thereby enabling machines to achieve functions that are difficult to achieve solely through direct programming (e.g., manually designing rules or manually constructing features). Using AI technology, machines can train the AI ​​model to be trained based on the collected training data. During training, the AI ​​model can automatically learn the features of the data and automatically update model parameters. After training, an AI model that can be used to process data of the same type as the training data can be obtained. In this application embodiment, the AI ​​model can also be referred to as a model, learning model, AI / ML model, or other names, and this application embodiment is not limited in this regard.

[0062] When using an AI model to perform text recognition processing on an image, the text region can be located in the image, and the text within that region can be recognized. As an example, the AI ​​model can use a neural network (e.g., a convolutional neural network) to extract features from an image containing characters, use a classifier to classify the image containing characters, and use the character with the highest confidence level as the text recognition result for each character in the image, thereby recognizing the text in the image. As another example, the AI ​​model can use a text region as input to the AI ​​model for feature extraction, and use the extracted features for decoding, thereby recognizing the text. For yet another example, the AI ​​model can employ an end-to-end text recognition model. This application does not limit this approach.

[0063] While AI models can achieve good text recognition results, their processing time is longer compared to traditional image processing algorithms. Therefore, considering both timeliness and processing efficiency, some possible implementations typically employ both traditional image processing algorithms and AI models to simultaneously perform text recognition on the first content.

[0064] The following will refer to Figure 2 An embodiment of the character recognition method of this application will be described.

[0065] In one possible implementation provided in the embodiments of this application, such as Figure 2 As shown in (a), after determining the first content, the terminal device can perform text recognition processing on the first content using simple OCR. Optionally, the terminal device can also simultaneously perform text recognition processing on the first content using complex OCR. As an example, simple OCR can use traditional image processing algorithms, while complex OCR can use AI models. Traditional image processing algorithms have a relatively fast processing speed, for example, a processing time of about 150 milliseconds, but low text recognition accuracy. In contrast, AI models have a relatively slow processing speed, for example, a processing time of about 300 milliseconds, but high text recognition accuracy.

[0066] Therefore, in one possible implementation, traditional image processing algorithms can be used simultaneously as simple OCR and AI models as complex OCR to perform text recognition on the first content. That is, both the traditional image processing algorithm and the AI ​​model begin text recognition on the first content concurrently. If the traditional image processing algorithm outputs a text recognition result, this result is directly used as the final text recognition result, and the AI ​​model's text recognition process is terminated. If the traditional image processing algorithm fails to output a text recognition result, the AI ​​model is waited to process the first content, and its output is ultimately used as the final text recognition result.

[0067] In one possible implementation, the terminal device can output the fastest text recognition result as the first content recognition result. Since there is a significant difference in processing speed between traditional image processing algorithms and AI models, the terminal device only needs to determine whether the traditional image processing algorithm produces a text recognition result. That is, if the traditional image processing algorithm can output a text recognition result, then that text recognition result is output as the first content recognition result. Figure 2As shown in (a), if the traditional image processing algorithm does not output a text recognition result, that is, the judgment result for "whether there is a recognition result when the traditional image processing algorithm performs text recognition on the first content" is "no", the terminal device can use the text recognition result output by the AI ​​model as the text recognition result of the first content.

[0068] In some scenarios, when performing text recognition processing on the primary content, traditional image processing algorithms may not output any text recognition results, i.e., they cannot output text. This may be because the primary content is a complex image. Traditional image processing algorithms can achieve fast and good text recognition for simple images, but their performance is poor for complex images, thus failing to output text recognition results.

[0069] In some scenarios, when performing text recognition on the first piece of content, traditional image processing algorithms may not output any text recognition results, i.e., they may not be able to output text, possibly because the first piece of content does not contain text. In such cases, even AI models cannot output text recognition results.

[0070] In one possible implementation provided in the embodiments of this application, such as Figure 2 As shown in (b) above, the terminal device can operate in accordance with... Figure 2 When operating on the first content in step (a), the terminal device can also use a layout detection model to determine the layout information of the first image. The layout detection model is used to determine the category of the content included in the first image and the area information corresponding to each category of content, and the category and the area information are the layout information.

[0071] As an example, the categories may include text categories, image categories, and QR code categories; the region information is used to indicate the position of content of a certain category in the first image. As an example, the terminal device may use the coordinate positions of the pixels of the block outline corresponding to the content to characterize the position of the content in the first image, wherein the coordinate system in which the coordinate data is located has the upper left corner of the screen as the origin and the lower right corner of the screen as the maximum coordinate value. Taking the image category as an example, the region information of the image category can indicate the position of the content of the image category (i.e., the image) in the first image; taking the text category as an example, the region information of the text category can indicate the position of the content of the text category in the first image.

[0072] In some cases, the first image may simultaneously contain multiple different categories of content. For example, a Weibo screenshot may include both a user-posted image and user-posted text, where the image is the image category and the text is the text category. Similarly, a movie screenshot may include both movie footage and subtitles, where the movie footage is the image category and the subtitles are the text category. Furthermore, a payment code screenshot may include both a QR code and merchant information (e.g., the text "You will pay XX Catering"), where the QR code is the QR code category and the merchant information is the text category.

[0073] In implementation, the layout detection model can be applied to text recognition processing. In one optional implementation, the terminal device can input a first image into the layout detection model to obtain text-type region information. Then, the terminal device determines the content to be recognized based on the region information. Finally, the terminal device can use the aforementioned traditional image processing algorithms and / or AI models to perform text recognition processing on the content and obtain the text recognition result.

[0074] In one alternative implementation, the user or technician can pre-set the size of the area (i.e., the unit area size) that the layout detection model processes, particularly the size of the text area. For example, the area size could indicate a text line or paragraph, etc. As an example, the layout detection model can determine the category of each text line on a line-by-line basis and output the position of each text line, such as... Figure 8 As shown, the layout detection model can output the category (i.e., text category) and position of text line 801, and the category (i.e., text category) and position of text line 802, etc. For example, the layout detection model can determine the category and position of each text paragraph. As another example, the layout detection model can output the category (i.e., text category) and position of text paragraph 810, and the category (i.e., text category) and position of text paragraph 820, etc. This layout detection model has a relatively fast processing speed, with a processing time of approximately 50 milliseconds. In one possible implementation, the aforementioned positions can be determined, for example, by the coordinate data of the four vertices of a rectangle that selects the corresponding content in image 800.

[0075] In implementation, this layout detection model can adopt... Figure 5The multi-task learning model mentioned above is a model that takes into account the correlation between different tasks and uses training data from multiple tasks to train it together. The two tasks performed by the layout detection model mentioned above (including the task of determining the category of content and the task of determining the area information corresponding to each category of content) are related, so a multi-task learning model is more suitable. The multi-task learning model will be described in detail in a later section, and will not be repeated here.

[0076] In one possible implementation provided in this application embodiment, the terminal device may also perform text recognition on the first content by simultaneously using a simple AI model as a simple OCR and a complex AI model as a complex OCR.

[0077] The distinction between simple and complex AI models can be based on their complexity. Generally, the more neural network layers (e.g., convolutional layers, pooling layers, fully connected layers, etc.) an AI model has, the more complex it is. Simple AI models typically process text faster than complex ones. Therefore, when both simple and complex AI models begin text recognition simultaneously, the simpler AI model will obtain the recognition result first, but its recognition performance may not be as good as the more complex model.

[0078] In one possible implementation, the terminal device can simultaneously utilize both a simple AI model and a complex AI model to perform text recognition on the first content. That is, both the simple and complex AI models begin text recognition simultaneously. If the simple AI model outputs a text recognition result after recognizing the first content, this result is directly used as the final text recognition result, terminating the complex AI model's text recognition process. If the simple AI model fails to output a text recognition result, the complex AI model is then waited to process the first content, and its output is ultimately used as the final text recognition result.

[0079] It can be seen that regardless of whether the image being processed is simple or complex, Figure 2In the text recognition methods shown in (a) and (b), the terminal device simultaneously starts two text recognition algorithms (i.e., simple OCR and complex OCR), and prioritizes the result of the faster algorithm (i.e., simple OCR) as the final text recognition result. Only when the faster algorithm (i.e., simple OCR) cannot output a text recognition result is the result of complex OCR used as the final text recognition result. This text recognition method can provide the user with text recognition results in a very short time when processing simple images; however, this method may have the following problems:

[0080] (1) Increase unnecessary computational load on terminal devices.

[0081] For simple images, the terminal device still needs to use two text recognition algorithms (i.e., simple OCR and complex OCR) to perform text recognition on the simple image simultaneously, which increases the unnecessary computational load on the terminal device. In other words, for simple images, the text recognition accuracy of simple OCR is usually sufficient to meet user expectations, so running complex OCR on simple images at the same time will generate unnecessary computational load.

[0082] (2) The accuracy of text recognition results is affected.

[0083] For complex images, simple OCR is faster at text recognition than complex OCR, but its accuracy is lower. When a terminal device only considers the recognition speed, simple OCR processes complex images faster than complex OCR, but the text recognition results may be inaccurate or even contain errors. In other words, for complex images, the accuracy of simple OCR is usually low, below user expectations, and fails to meet user needs for text recognition. Therefore, even when simple OCR can output text recognition results for complex images, the results often do not meet user expectations (e.g., recognizing only 10% of the text in the complex image), thus reducing the user experience.

[0084] Based on this, embodiments of this application propose a text recognition method. This method first determines the image evaluation level of the image to be processed, and then determines a more suitable text recognition method for the image based on the image evaluation level. The image evaluation level mentioned in this application is information evaluating an image from the perspective of whether the text is easily recognizable. If the text in an image is easily recognizable, the image evaluation level is considered high; conversely, if the text in an image is not easily recognizable, the image evaluation level is considered low. The method can employ text recognition algorithms of varying complexity depending on the image evaluation level. If the image evaluation level is high, a low-complexity text recognition algorithm (i.e., a simple text recognition algorithm), such as a traditional text recognition algorithm or a simple AI model, is used to recognize the text in the image. For ease of description, low-complexity or simple text recognition algorithms can be referred to as simple OCR. If the image evaluation level is relatively low, a high-complexity text recognition algorithm (i.e., a complex text recognition algorithm) is used, such as an AI model or a complex AI model. For ease of description, a high-complexity text recognition algorithm or a complex text recognition algorithm can be called a complex OCR, while a low-complexity text recognition algorithm or a simple text recognition algorithm can be called a simple OCR.

[0085] The following will refer to Figure 3 A flowchart illustrating the steps of a text recognition method according to an embodiment of this application is described in detail.

[0086] Step S310: Display the first content, which is an image containing text.

[0087] The first content may indicate an area of ​​interest to the user; that is, the first content may include an area from which the user expects text recognition to be performed. In implementation, the user may select the first content from a first image already displayed on the terminal device. As an example, the first content may be the first image by default. That is, the method may use the first image displayed on the first user interface as the first content.

[0088] Specifically, a user can select first content from a first image through a content selection operation. This content selection operation refers to determining the first content using a user's body part (e.g., a finger) or input device (e.g., a stylus, keyboard, mouse, etc.). As an example, the terminal device can use a body part or input device to obtain the start and end positions of the first content, and then use the start and end positions as vertices of a polygon (e.g., a rectangle) to define a polygonal region as the first content.

[0089] As an example, if the body part or input device continues to input the starting position in the first image and the continuously input path constitutes a regular or irregular area, then the regular or irregular area can be determined as the first content.

[0090] As an example, after detecting touch input to the first image, the terminal device can determine the initial touch position and the end touch position of the touch input. In implementation, the touch can include contact touch and non-contact touch (e.g., hover touch). Furthermore, the touch can include user limb touch (e.g., finger touch) and touch device touch (e.g., stylus touch). Subsequently, the terminal device can determine a rectangular area (i.e., the first content) formed by the initial touch position and the end touch position. As another example, the terminal device can also use simultaneously received touch positions as the initial touch position and the end touch position respectively to determine the first content. The above are merely examples of determining the first content, and this application is not limited thereto.

[0091] As an example, the first image may be an image selected by the user from a stored image library, an image captured by the user using a camera, a screenshot obtained after the user performs a screenshot operation, or an image downloaded or received by the user via the network. This application does not limit the scope of the image.

[0092] Optionally, the terminal device can also determine multiple text contents on the same image. As an example, a user can determine a first content on a first image. After the terminal device performs text recognition on the first content, the user can determine a second content on the first image. The terminal device then performs text recognition on the second content. Since the first content is different from the second content, the recognition method for performing text recognition on the first content may be different from the recognition method for performing text recognition on the second content.

[0093] Step S320: Receive the text recognition operation performed by the user on the first content.

[0094] In one optional implementation, the text recognition operation can be a user-triggered operation on the text recognition control. Illustratively, the triggering operation includes any one or more combinations of click, swipe, press, and long press operations. In other possible implementations, the text recognition operation can also be a voice-based operation. For example, the terminal device receives a voice signal input by the user, parses the voice signal to obtain the voice content, and when the voice content contains keywords matching the text recognition control, the terminal device determines that the text recognition control has been triggered, i.e., the terminal device receives a trigger operation on the first control. Furthermore, users can also perform text recognition operations through combined operations (e.g., long press, double click, triple click, two-finger press).

[0095] Step S330: In response to the text recognition operation, determine the image evaluation level corresponding to the first content.

[0096] In this application embodiment, the image evaluation level is evaluated only from the perspective of whether the text in the image is easily recognizable. That is, if the text in the image is easily recognizable, the image evaluation level is considered to be relatively high, and simple OCR (e.g., traditional text recognition algorithms or simple AI models) can be used to recognize the text in the image; conversely, if the text in the image is not easily recognizable, the image evaluation level is considered to be relatively low, and complex OCR (e.g., AI models or complex AI models) can be used.

[0097] Optionally, the evaluation criteria for an image's rating level may include, but are not limited to, one or more of the following: whether the text in the image is a printed font, whether the text lines in the image are horizontal, whether there are curved curves in the image, and whether the font in the image is clear. In practice, if the font in the image is a printed font, the text lines in the image are horizontal, and / or there are no curved curves in the image, then the image's rating level is considered high; conversely, if the font in the image is a non-printed font, the text lines in the image are not horizontal, and / or there are curved curves in the image, then the image's rating level is considered low. Based on the above settings, screenshot images typically have a higher rating level, while captured images typically have a lower rating level.

[0098] Furthermore, the image evaluation level can be evaluated according to its degree. For example, the image evaluation level may include high, medium, and low; or, for example, excellent, good, medium, and poor; or, for example, a first image evaluation level, a second image evaluation level, and a third image evaluation level, etc. As another embodiment, the image evaluation level can be quantified into a specific numerical value representing image quality. This application does not limit this. It should be noted that the implementation details of determining the image evaluation level can be found in the relevant descriptions of the following embodiments, and will not be described here.

[0099] Step S340: Use the text recognition method corresponding to the image evaluation level to recognize the first content and obtain the text recognition result.

[0100] When the image evaluation level indicates a simple image, a first text recognition method can be used to recognize the first content. The first text recognition method can indicate simple OCR, which is typically fast but has low text recognition accuracy. In one possible implementation, the first text recognition method can indicate a method that recognizes text using traditional image processing algorithms. For example, the image processing algorithm can employ operations such as text region detection, text line merging, single character segmentation, and single character recognition to recognize text in an image. In another possible implementation, the first text recognition method can also indicate a relatively simple AI model, which has a simple structure and therefore a fast processing speed.

[0101] In an optional embodiment, when the image evaluation level indicates a complex image, a second text recognition method can be used to recognize the first content. The second text recognition method can indicate complex OCR, which typically has a slower processing speed but higher text recognition accuracy. In practice, the second text recognition method can indicate a method that uses a complex AI model to recognize text. The complex AI model can indicate using a trained AI model to perform text recognition.

[0102] In summary, after receiving a text recognition operation for the first content, the terminal device first determines the image evaluation level of the first content, and then uses the text recognition algorithm corresponding to the image evaluation level to perform text recognition on the first content, thereby enabling more accurate recognition of the text in the first content and reducing unnecessary computation.

[0103] Figure 4 An illustration of an image evaluation level according to an embodiment of this application is shown. For ease of understanding, reference will be made below. Figure 4 Please specify the image evaluation level involved in this application. Figure 4 In the image, (a) and (b) represent two images with different image evaluation levels. For example... Figure 4 As shown in (a), this image is a screenshot of a user interface containing text content. It can be seen that the text in the image is arranged and sized regularly; therefore, this image has a high image rating. As an example, this screenshot can be classified as a first-level image. In comparison, Figure 4 As shown in (b), the image is a photograph taken by the user using the camera of the terminal device. The text in the photograph is irregularly arranged and of irregular size, therefore the image rating level is low. As an example, the photographed image can be judged as a second image rating level.

[0104] To obtain the image rating level corresponding to the first content, embodiments of this application may employ an image rating level determination model to determine the first content. The image rating level determination model may be a model including a neural network structure. Before determining the first content, embodiments of this application may train the image rating level determination model using training data, wherein the training data includes multiple training images and the image rating level corresponding to each training image.

[0105] Optionally, the image evaluation level determination model may include, for example, Figure 5 The multi-task learning model 510 shown in (a) or as shown in the figure. Figure 5 The single-task learning model shown in (b) will be referred to below. Figure 5 Describe it.

[0106] like Figure 5 As shown in (a), the multi-task learning model 510 performs step S330 to determine the first image evaluation level corresponding to the first content.

[0107] In implementation, the text recognition method according to the embodiments of this application can input a first image as input information into a multi-task learning model 510. After processing the first image, the multi-task learning model 510 can output the category information, region information, and image evaluation level of the first image. The multi-task learning model is a model that takes into account the correlation between different tasks and uses training data from multiple tasks to train it together.

[0108] Combination Figure 6 Specifically, the multi-task learning model can utilize shared convolutional layers 602 to obtain shared features. After obtaining the shared features, different results are output depending on the set task. When the input image 601 is input into the already trained multi-task learning model, the trained multi-task learning model can output the following three results:

[0109] (1) The category of the content contained in the input image 601.

[0110] Input image 601 contains content categorized as both text and image. Therefore, the multi-task learning model can output "text" and "image" to indicate that input image 601 contains content categorized as text and image, respectively. Alternatively, the multi-task learning model can output identifiers indicating the categories of content contained in input image 601. For example, outputting "0" indicates that input image 601 contains content categorized as text, and outputting "1" indicates that input image 601 contains content categorized as image; or, for example, outputting "01" indicates that input image 601 contains content categorized as both text and image.

[0111] (2) The position of a certain category of content contained in the input image 601 within the input image 601.

[0112] The text category content contained in input image 601 is located below input image 601, so the multi-task learning model can output the position of rectangle 610 to indicate the position of the text category content contained in input image 601 within input image 601. In addition, the image category content contained in input image 601 is located above input image 601, so the multi-task learning model can output the position of rectangle 620 to indicate the position of the image category content contained in input image 601 within input image 601.

[0113] (3) Image evaluation level of input image 601.

[0114] The input image 601 is a screenshot image. This multi-task learning model can output evaluation level information for the input image 601, such as "high" or "excellent".

[0115] Optionally, the above three results can be output simultaneously or not simultaneously according to preset conditions. For example, the multi-task learning model can output only the image evaluation level of the input image 601. In addition, in one possible implementation, the trained multi-task learning model can also be set to output only text-related results. For example, the trained multi-task learning model can be set to output: the judgment result of whether the input image 601 contains text category content (e.g., output "Yes / Y / Yes / 1" or output "No / N / No / 0"); if the input image 601 contains text category content, the multi-task learning model can also output the position of the text category content in the input image 601 and the image evaluation level of the input image 601.

[0116] In implementation, such as Figure 5 The multi-task learning model 510 shown can be implemented as follows: Figure 2 This model is an improvement upon the layout detection model described above. In practice, this multi-task learning model can output category information and region information, as well as an image evaluation level. The image evaluation level is information evaluating the image based on how easily the text can be recognized. For example, naturally captured images are complex, therefore, the image evaluation level is low; screenshot images are simple, therefore, the image evaluation level is high.

[0117] During training, the various parameters in the multi-task learning model can be adjusted using the correspondences in the training data. The training data includes training images and their corresponding label information; the correspondences in the training data refer to the correspondence between the training images and their corresponding label information. The label information of the training images may include, for example, the category of the content contained in the training image (i.e., category information), the location of each category of content in the training image (i.e., region information), and the image evaluation level of the training image.

[0118] In one possible implementation of the training process, a phased training approach can be used to train the multi-task learning model. Considering the correlation between category information and region information, parameters related to the category task and the location task in the neural network layers can be adjusted first. Subsequently, while keeping these parameters unchanged, the parameters related to the image evaluation level in the neural network layers are adjusted until the preset requirements are met. In another possible implementation of the training process, the correspondence in the training data can be used to simultaneously adjust all parameters in the multi-task learning model until the preset requirements are met. In practical use, the multi-task learning model trained using a phased training approach does not affect the results of the different tasks output separately, and the accuracy is higher.

[0119] The preset requirement indicates that the calculated loss function value after each update of model parameters during training is less than a predetermined threshold. The loss function value refers to the difference between the predicted result calculated using the loss function and the actual result. During training, the multi-task learning model can adjust each model parameter using the correspondence in the training data, and after updating the model parameters, it again uses the training data to calculate a new loss function value. As training continues, the obtained loss function value will continuously decrease until it is less than the predetermined threshold, at which point the training process ends, meaning the multi-task learning model has completed training. In implementation, the loss function can be a cross-entropy loss function or a loss function suitable for classification networks.

[0120] In practice, the image evaluation level output by the multi-task learning model may be incorrect. For example, if the input image has a "low" image evaluation level (e.g., a complex image), the multi-task learning model may incorrectly output a "high" image evaluation level; conversely, if the input image has a "high" image evaluation level (e.g., a simple image), the multi-task learning model may incorrectly output a "low" image evaluation level. In the case of an input image with a "high" image evaluation level, the terminal device can accept, to some extent, the multi-task learning model incorrectly outputting a "low" image evaluation level. This is because even if the terminal device subsequently uses complex OCR to perform text recognition on the simple image based on the "low" image evaluation level, it will not lead to a decrease in text recognition accuracy, only a possible increase in processing time. However, in the case of an input image with a "low" image evaluation level, if the multi-task learning model incorrectly outputs a "high" image evaluation level, the terminal device may use simple OCR to perform text recognition on the complex image based on the "high" image evaluation level, resulting in low text recognition accuracy or even failure to recognize the text. Therefore, the terminal device cannot accept the multi-task learning model incorrectly outputting a "high" image evaluation level.

[0121] Based on this, in one possible implementation of the training process, different preset thresholds can be set for the loss function value for different training data. Specifically, if the training image is a "high" image evaluation level image (e.g., a simple image), the preset threshold can be appropriately lowered. That is, the multi-task learning model continues training after the loss function value reaches the preset threshold until the loss function value is lower than the original preset threshold before ending the training. This will result in a higher accuracy rate for images judged as "high" image evaluation levels. Therefore, in implementation, the multi-task learning model will be less likely to judge the image evaluation level as "high". If the training image is a "low" image evaluation level image (e.g., a complex image), the preset threshold can be appropriately raised. That is, the multi-task learning model ends training before it is fully trained, even if its loss function value has reached a value higher than the original preset threshold. This will result in a lower accuracy rate for images judged as "low" image evaluation levels. Therefore, in implementation, the multi-task learning model will be more likely to judge the image evaluation level as "low".

[0122] Based on the above description of the multi-task learning model training process, it can be seen that the embodiments of this application can obtain a multi-task learning model by simply improving the existing layout detection model and further training it. In this way, the terminal device can meet both existing needs and new needs generated by the terminal device using only the multi-task learning model, reducing the burden on the terminal device.

[0123] like Figure 5 As shown in (b), the single-task learning model 520 executes step S330 to determine the first image evaluation level corresponding to the first content.

[0124] In implementation, the terminal device may use only the first content or the first image as input information for the single-task learning model 520. After processing the first content or the first image, the single-task learning model 520 outputs an image evaluation level. The single-task learning model 520 refers to a neural network model that performs only a single task. In implementation, the single-task learning model 520 may be trained using only the training data corresponding to that single task. That is, the single-task learning model 520 may be trained using only the training image and the corresponding image evaluation level label information, adjusting the model parameters until the loss function value is less than a predetermined threshold.

[0125] In one possible implementation, after selecting a first image, the user can manually input their image rating for that first image. As an example, the user can select a screenshot from a gallery as the first image and then input their image rating for that first image using relevant controls on the first user interface (e.g., an image rating control). For example, for a screenshot, the user can manually input an image rating of "high".

[0126] In one possible implementation, the user can first manually input an image rating level and then select the first image. As an example, the user can input an image rating of "low" using the relevant controls and then select an image captured by the camera as the first image.

[0127] In summary, the text recognition method provided in this application embodiment first determines the image evaluation level of the image, and then selects the corresponding text recognition method based on the image evaluation level, thereby improving the accuracy of text recognition and reducing unnecessary computation. Furthermore, the text recognition method can upgrade existing layout detection models to obtain a multi-task learning model that can simultaneously output layout detection results and image evaluation levels, thereby enabling the reuse of internal algorithms to meet new demands and reducing the burden on terminal devices.

[0128] Figure 7 A flowchart illustrating the steps of a method for judging character recognition results according to an embodiment of this application is provided. Figure 7 Before the method shown, first combine Figure 8 The concepts involved in this embodiment will be introduced first. For ease of subsequent understanding, the concepts will be explained here. Figure 8 Describe it.

[0129] like Figure 8 As described above, image 800 may include a first text block 810, a second text block 820, and a third text block 830. Figure 8 In this text, both "*" and "-" represent characters. "*" indicates characters successfully recognized by the character recognition algorithm, while "-" indicates characters not recognized. In the example where the first text block 810 is used as the text region for character recognition, the first text block 810 includes regions 801, 802, 803, 804, and 805. Since the characters in regions 801, 802, and 803 have been successfully recognized by the character recognition algorithm, these regions are called recognized regions. Furthermore, since the characters in regions 804 and 805 have not been successfully recognized by the character recognition algorithm, these two regions 804 and 805 can be called unrecognized regions.

[0130] (1) Intersection over union (IOU).

[0131] The intersection-union ratio (IUU) indicates the overlap rate between the identified region and the text region. In practice, it can be determined using the intersection and union of the identified region and the text region. Specifically, the intersection of the identified region and the text region indicates the area of ​​the overlapping region. If the identified region is part of the text region, the intersection of the identified region and the text region is the area of ​​the identified region. If the identified region and the text region do not overlap at all, the intersection of the identified region and the text region is 0.

[0132] The union of the identified region and the text region indicates the total area of ​​the region, whether the identified region and the text region overlap or not. If the identified region is a part of the text region, then the union of the identified region and the text region is the text region. If the identified region and the text region do not overlap at all, then the union of the identified region and the text region is the sum of the areas of the text region and the identified region.

[0133] against Figure 8 The first text block 810 in the text has identified regions 801, 802, and 803. Therefore, the intersection of the identified regions and the text regions is the sum of the areas of regions 801, 802, and 803, and the union of the identified regions and the text regions is the area of ​​the first text block 810 (the areas of regions 801, 802, 803, 804, and 805). Therefore, the intersection-union ratio of the identified regions (including regions 801, 802, and 803) with the first text block is the overlap rate between the identified regions and the first text block 810, which is 62.5%. Similarly, for... Figure 8The intersection-union ratio of the identified regions (including regions 806 and 807) with the second text block 820 is the overlap rate between the identified regions and the second text block 820, which is 100%. For... Figure 8 The third text block 830 in the text does not contain an identified region. Therefore, the intersection-union ratio between the identified region and the third text block 830 is 0.

[0134] As one possible implementation, the identified region may include not only a portion of the text region but also other parts. In this case, the intersection of the identified region and the text region remains unchanged, but the union of the identified region and the text region needs to be increased by the area of ​​the aforementioned partial regions. Figure 8 For example, if the text region is the first text block, the identified regions include regions 801, 802, and 803, as well as regions 806 and 807. The intersection of the identified regions and the text region is still the sum of the areas of regions 801, 802, and 803. However, the union of the identified regions and the text region is the sum of the areas of the first text block 810 and the second text block 820. Therefore, the intersection-union ratio can be approximately 41.6%.

[0135] (2) Text completeness.

[0136] Text completeness indicates the ratio of the area of ​​the identified region to the total area of ​​the text region in each text region.

[0137] by Figure 8 Taking the first text block 810 as an example, when the first text block 810 is considered a text region, the identified regions are regions 801, 802, and 803. Therefore, the text integrity can be indicated by the ratio of the area of ​​regions 801, 802, and 803 to the area of ​​the first text block 810, which is 62.5%. For... Figure 8 The second text block 820, in which text integrity can indicate the ratio of the area of ​​the identified region (including region 806 and region 807) to the area of ​​the second text block 820, i.e., 100%. For... Figure 8 The third text block 830 in the text does not have an identified region, therefore, the text integrity of the third text block 830 is 0.

[0138] As one possible implementation, the identified region may include not only a portion of the text area but also other parts. Figure 8For example, if the text region is the first text block 810, the identified regions include regions 801, 802, and 803, as well as regions 806 and 807. In this case, the text integrity is still the ratio of the area of ​​regions 801, 802, and 803 to the area of ​​the first text block 810, which is 62.5%.

[0139] As one possible implementation, the terminal device may use the minimum text completeness among the text completeness values ​​of each text block within the image 800 as the text completeness value of the image 800.

[0140] Based on the above explanations of intersection-union ratio and text completeness, the following will combine... Figure 7 This application describes a determination method provided by an embodiment. The method includes steps S710 and S720:

[0141] Step S710: Determine whether the first character recognition result meets the preset requirements.

[0142] After obtaining the first character recognition result in step S340, step S710 is executed to determine whether the first character recognition result meets the preset requirements. If the first character recognition result meets the preset requirements, the first character recognition result is output. If the first character recognition result does not meet the preset requirements, step S720 is executed.

[0143] In some embodiments, the preset requirements in step S710 may include: the preset requirements for the intersection-union ratio (IUGR) and / or the preset requirements for text integrity. In some embodiments, during the determination process in step S710, the determination of the IUGR may be performed first; after determining that the IUGR meets the preset requirements, the determination of text integrity may be performed, as described below.

[0144] In one possible implementation, the preset requirement includes an intersection-union ratio (IUU) between the text recognition region corresponding to the first text recognition result and the first content region of the first content being greater than a first preset threshold. For example, if it is determined that the first text recognition result meets the preset requirement—that is, the IUU between the text recognition region corresponding to the first text recognition result and the first content region of the first content is greater than the first preset threshold—then the first text recognition result can be provided to the user as the text recognition result of the first content. The first preset threshold can be a value preset by a technician, and this application does not impose any restrictions. For example, the first preset threshold can be set to 0, meaning that as long as text is recognized from the first content, the preset requirement is met. If it is determined that the first text recognition result does not meet the preset requirement—that is, if the IUU is not greater than (for example, less than or equal to) the first preset threshold—then step S720 is executed. It can be seen that the terminal device can use the IUU as a preset requirement, thereby enabling it to quickly determine whether text is recognized in the first text recognition result.

[0145] In one possible implementation, the preset requirement may further include that the text completeness determined by the first character recognition result and the information of the first content region to which the first content belongs is greater than a second preset threshold. In practice, if the first character recognition result meets the preset requirement, that is, the text completeness determined by the first character recognition result and the information of the first content region to which the first content belongs is higher than the second preset threshold, then the first character recognition result can be provided to the user as the character recognition result of the first content. The second preset threshold can be a value preset by a technician, and this application does not impose any restrictions. If it is determined that the first character recognition result does not meet the preset requirement, that is, if it is determined that the text completeness of the first character recognition result is not greater than (for example, less than or equal to) the second preset threshold, then step S720 is executed. For example, when the second preset threshold is set to 0.8, the text completeness of the first character recognition result is the ratio of the area of ​​the character recognition region of the first text block 810 to the area of ​​the first text block 810, i.e., 62.5%. It can be seen that if the first character recognition result does not meet the preset requirement, then step S720 can be executed. It can be seen that the terminal device can use text integrity as a preset requirement, so as to evaluate the recognition effect of the first text recognition result. If the recognition effect does not meet the requirements or does not achieve the expected effect, step S720 can be executed.

[0146] In step S720, the second character recognition result identified by the second character recognition algorithm is taken as the character recognition result.

[0147] If step S710 determines that the first character recognition result does not meet the preset requirements, step S720 can be executed. That is, if the first character recognition result obtained using simple OCR does not meet the expected requirements or does not achieve the expected effect, the second character recognition result obtained using complex OCR can be used as the character recognition result. In implementation, the terminal device can display the second character recognition result on the screen, or the terminal device can verbally announce the second character recognition result, etc.

[0148] In summary, after determining that simple OCR is used to recognize text, the terminal device in this embodiment can use the intersection-union ratio (IUGR) to judge the text recognition result and determine whether the text has been recognized. If the text has been recognized, the method can further determine the text recognition effect using text completeness. If the text recognition effect is not as expected (i.e., below a preset threshold), the recognition result of complex OCR can be used. In this process, the method can use measurement indicators (including text completeness and / or IUGR) to quantify the text recognition result, making the judgment more reliable.

[0149] Figure 9 This application provides an exemplary text recognition method. This method not only determines the text recognition algorithm (simple OCR and complex OCR) based on the complexity of the image, but also further verifies the text recognition results output by simple OCR. In this process, to quantify the judgment, a quantitative indicator (i.e., text completeness) is introduced, making the judgment operation more reliable. Figure 9 As shown, the method includes the following steps:

[0150] Step S900: Acquire the first image. That is, the terminal device can use the first image as a text recognition area.

[0151] Step S910: Determine the complexity of the first image. The complexity of the first image can be indicated by the image evaluation level mentioned above, therefore, it will not be described in detail here. If step S910 determines that the first image is a simple image, then proceed to step S920; if step S910 determines that the first image is a complex image, then proceed to step S940.

[0152] Step S920: Use simple OCR to perform text recognition on the first image and obtain the first text recognition result. The simple OCR can be a traditional image processing algorithm or a simple AI model as described above.

[0153] Step S940: Use complex OCR to perform text recognition on the first image to obtain a second text recognition result, wherein the complex OCR can be the complex AI model described above.

[0154] Step S930: Determine if the text completeness is greater than a second preset threshold. The text completeness can be determined using the first character recognition result and the first content region to which the first content belongs. If the text completeness is greater than the second preset threshold, the first character recognition result can be displayed on the screen. Conversely, if the text completeness is less than the second preset threshold, step S940 can be used to perform character recognition on the first image using complex OCR, and the second character recognition result can be displayed on the screen.

[0155] In summary, the text recognition method according to the embodiments of this application can first determine the complexity of the image to be processed, and then determine the text recognition method based on the complexity, thereby ensuring recognition speed and improving recognition accuracy, while reducing the computational load of the terminal device. Furthermore, to further ensure the accuracy of text recognized by simple text recognition methods, the text recognition results output by simple text recognition algorithms can be further verified. In this process, to quantify the judgment, a measurement index (i.e., text completeness) is introduced, making the judgment operation more reliable.

[0156] Figure 10 This diagram illustrates the structure of a terminal device 100.

[0157] Terminal device 100 may include at least one of the following: mobile phone, foldable electronic device, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device, or smart city device. This application embodiment does not impose any special limitation on the specific type of terminal device 100.

[0158] Terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) connector 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0159] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0160] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0161] The processor can generate operation control signals based on the instruction opcode and timing signals to control instruction fetching and execution. In this embodiment, the processor 110 can be used to execute implementations. Figure 3 , Figure 7 and Figure 9The computer instructions for the method shown. In this process, the processor may also include, for example... Figure 5 and Figure 6 Computer instructions for the image rating level determination model.

[0162] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. This memory can store instructions or data that the processor 110 has used or that are used frequently. If the processor 110 needs to use the instruction or data, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0163] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 110 can connect to modules such as touch sensors, audio modules, wireless communication modules, displays, and cameras through at least one of these interfaces.

[0164] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0165] The charging management module 140 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger.

[0166] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, display 194, camera 193, and wireless communication module 160, etc.

[0167] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0168] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0169] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0170] The wireless communication module 160 can provide solutions for wireless communication applications on the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth Low Energy (BLE), ultra-wideband (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0171] In some embodiments, antenna 1 of terminal device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling terminal device 100 to communicate with networks and other electronic devices via wireless communication technology. This wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0172] Button 190 may include a power button, volume buttons, etc. Button 190 may be a mechanical button or a touch button. Terminal device 100 may receive button input and generate key signal inputs related to user settings and function control of terminal device 100.

[0173] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0174] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0175] Camera 193 (either a front-facing camera or a rear-facing camera, or a single camera that can function as both) is used to capture images or videos. Typically, camera 193 may include a photosensitive element such as a lens assembly and an image sensor. The lens assembly includes multiple lenses (convex or concave lenses) for collecting light signals reflected from the object being photographed and transmitting these signals to the image sensor. The image sensor generates a raw image of the object being photographed based on the light signals. In this embodiment, terminal device 100 can invoke camera 193 to acquire a captured image, which is determined to be a complex image.

[0176] Terminal device 100 can implement display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0177] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include one or more display screens 194.

[0178] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the terminal device 100. The terminal device 100 can support one or more SIM card interfaces. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0179] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card, or music, video, and other files can be transferred from the electronic device to the external storage card.

[0180] Internal memory 121 can be used to store computer executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of terminal device 100 (such as audio data, phone book, etc.). In addition, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional methods or data processing of terminal device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor.

[0181] Terminal device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0182] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0183] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music through the speaker 170A or output audio signals for hands-free calling.

[0184] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 100 answers a phone call or voice message, the receiver 170B can be brought close to the listener's ear to hear the voice.

[0185] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal device 100 may be equipped with at least one microphone 170C. In some embodiments, terminal device 100 may be equipped with two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal device 100 may be equipped with three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0186] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0187] Sensor module 180 may include the following sensors:

[0188] Pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Terminal device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to display screen 194, terminal device 100 detects the intensity of the touch operation based on pressure sensor 180A. Terminal device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In this embodiment, terminal device 100 can determine the area pressed by the user's two fingers based on the detection signal from pressure sensor 180A and define this area as the first content.

[0189] The gyroscope sensor 180B can be used to determine the motion attitude of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for image stabilization.

[0190] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates altitude based on the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0191] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. When the electronic device is a foldable electronic device, the magnetic sensor 180D can be used to detect the folding or unfolding of the electronic device, or the folding angle.

[0192] The 180E accelerometer can detect the magnitude of acceleration in various directions (typically three axes) of the terminal device 100. When the terminal device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0193] A distance sensor 180F is used to measure distance. The terminal device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, the terminal device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0194] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The terminal device 100 emits infrared light outward through the LED. The terminal device 100 uses the photodiode to detect infrared reflected light from nearby objects. When the intensity of the detected reflected light is greater than a threshold, it can be determined that there is an object near the terminal device 100. When the intensity of the detected reflected light is less than the threshold, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 may use the proximity sensor 180G to detect when a user holds the terminal device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and screen locking.

[0195] The ambient light sensor 180L can be used to sense ambient light intensity. The terminal device 100 can adaptively adjust the brightness of its display screen 194 based on the sensed ambient light intensity. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking photos. The ambient light sensor 180L can also work in conjunction with the proximity sensor 180G to detect whether the terminal device 100 is obstructed, such as when the electronic device is in a pocket. When obstruction or being in a pocket is detected, some functions (such as touch functionality) can be disabled to prevent accidental operation.

[0196] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0197] Temperature sensor 180J is used to detect temperature. In some embodiments, terminal device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature detected by temperature sensor 180J exceeds a threshold, terminal device 100 reduces processor performance to reduce power consumption of the electronic device for thermal protection. In other embodiments, when the temperature detected by temperature sensor 180J is below another threshold, terminal device 100 heats battery 142. In still other embodiments, when the temperature is below yet another threshold, terminal device 100 may boost the output voltage of battery 142.

[0198] Touch sensor 180K, also known as a "touch device," can be disposed on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of terminal device 100, in a different location than display screen 194. In this embodiment, touch sensor 180K can detect a user's touch on a control used for indicating text recognition and transmit the detected touch operation to an application sensor.

[0199] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can analyze the voice signals based on the vibration signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality.

[0200] The software system of terminal device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of terminal device 100.

[0201] Figure 11 This is a software structure block diagram of the terminal device 100 according to an embodiment of this application.

[0202] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime (ART) and native C / C++ libraries, the Hardware Abstraction Layer (HAL), and the kernel layer.

[0203] The application layer can include a series of application packages.

[0204] like Figure 11As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS. In this embodiment, the application package may also include a specific application that performs the text recognition method.

[0205] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0206] like Figure 11 As shown, the application framework layer may include a window manager, content provider, view system, resource manager, notification manager, activity manager, input manager, etc.

[0207] The window manager provides Window Manager Service (WMS), which can be used for window management, window animation management, surface management, and as a relay station for the input system.

[0208] A content provider is used to store and retrieve data, making this data accessible to applications. This data may include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, etc. In this embodiment, the content provider can be used to store and retrieve data related to user content selection operations on a first image or text recognition operations performed on first content.

[0209] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images. In embodiments of this application, the display interface may include, for example... Figure 1 The interface in (a), (b), (c) or (d) of the document.

[0210] The notification manager allows applications to display notification information in the status bar. It can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also appear as an icon or scrolling text in the system's top status bar, such as notifications from background applications, or as a dialog window on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, vibrating the electronic device, or flashing indicator lights. In this embodiment, after the terminal device 100 obtains the text recognition result, the notification manager can emit an alert sound to notify the user that text recognition has been completed, allowing the user to check promptly.

[0211] The input manager can provide an Input Manager Service (IMS), which can be used to manage system inputs, such as touch input, keypad input, and sensor input. The IMS retrieves events from input device nodes and, through interaction with the WMS, distributes the events to appropriate windows. In this embodiment, the IMS provided by the input manager can be used to manage user input for selecting first content and user input for performing character recognition on the first content.

[0212] The Android runtime consists of the core libraries and the Android runtime itself. The Android runtime is responsible for converting source code into machine code. The Android runtime primarily employs ahead-of-time (AOT) compilation and just-in-time (JIT) compilation techniques.

[0213] The core library primarily provides basic Java class library functionalities, such as libraries for fundamental data structures, mathematics, I / O, tools, databases, and networking. It also provides APIs for users to develop Android applications.

[0214] Native C / C++ libraries can include multiple functional modules. Examples include: surface manager, media framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0215] The Surface Manager manages the display subsystem and provides 2D and 3D layer blending for multiple applications. The Media Framework supports playback and recording of various common audio and video formats, as well as still image files. The Media Library supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. OpenGL ES provides drawing and manipulation of 2D and 3D graphics in applications. SQLite provides a lightweight relational database for applications on terminal devices.

[0216] The Hardware Abstraction Layer (HAL) runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to the upper layers.

[0217] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0218] The following example, using a scene of capturing a photograph, illustrates the workflow of the software and hardware of the terminal device 100.

[0219] When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a single click, the control corresponding to the single click is as follows: Figure 1 Taking the text recognition control in (d) as an example, the text recognition application calls the interface of the application framework layer to start the text recognition service, and then executes the text recognition method of this application embodiment.

[0220] An embodiment of this application provides a terminal device, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions.

[0221] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0222] Embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0223] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0224] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0225] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0226] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0227] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0228] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0229] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0230] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0231] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0232] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A character recognition method, applied to a terminal device, characterized in that, The method includes: Display first content, which is an image containing text, and the first content is determined in response to a user's content selection operation on the first image; Receive text recognition operations performed by the user on the first content; In response to the text recognition operation, an image evaluation level corresponding to the first content is determined, the image evaluation level being used to indicate whether the image is easily recognized as text; The first content is identified using the text recognition algorithm corresponding to the image evaluation level to obtain the text recognition result. The image evaluation level includes a first image evaluation level and a second image evaluation level. The image with the first image evaluation level includes a screenshot image, which includes text that is regularly arranged and regularly sized. The image with the second image evaluation level includes a photograph image, which includes text that is irregularly arranged and irregularly sized. The step of using the text recognition algorithm corresponding to the image evaluation level to recognize the first content and obtain the text recognition result includes: When the image evaluation level is the first image evaluation level, the first text recognition algorithm is used to recognize the first content to obtain the first text recognition result, or... When the image evaluation level is the second image evaluation level, the first content is recognized by the second character recognition algorithm to obtain the second character recognition result. The second character recognition result is determined as the final output character recognition result. The algorithm complexity of the first character recognition algorithm is lower than that of the second character recognition algorithm, or the recognition speed of the first character recognition algorithm is faster than that of the second character recognition algorithm.

2. The method according to claim 1, characterized in that, When the image evaluation level is the first image evaluation level, after using a first character recognition algorithm to recognize the first content and obtaining the first character recognition result, the method further includes: Determine whether the first character recognition result meets the preset requirements; If the first character recognition result meets the preset requirements, the first character recognition result is determined as the final output character recognition result; If the first text recognition result does not meet the preset requirements, the second text recognition algorithm is used to recognize the first content, obtain the second text recognition result, and determine the second text recognition result as the final output text recognition result.

3. The method according to claim 2, characterized in that, The preset requirements include that the intersection-union ratio of the text recognition region corresponding to the first text recognition result and the region of the first content is greater than a first preset threshold, and / or that the text integrity of the text recognition region corresponding to the first text recognition result and the region of the first content to which the first content belongs is greater than a second preset threshold.

4. The method according to any one of claims 1 to 3, characterized in that, Determining the image evaluation level corresponding to the first content includes: The first content is input into the image evaluation level determination model to obtain the image evaluation level corresponding to the first content. The image evaluation level determination model is a model trained based on training data. The training data includes multiple training images and the image evaluation level corresponding to each training image. The image evaluation level determination model includes a neural network structure.

5. The method according to claim 4, characterized in that, The image evaluation level determination model is a multi-task learning model, and the image evaluation level determination model is also used to obtain the category of the first content and the region information corresponding to the category.

6. The method as described in claim 1, characterized in that, Before displaying the first content, the method further includes: In response to the user's content selection operation for the first image, a first content is determined, wherein the first content is part or all of the content in the first image.

7. A terminal device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 6 when executing the instructions.

8. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes instructions that instruct a processor to perform the steps of the method according to any one of claims 1 to 6.

Images