Text recognition method and electronic device

Abstract

Embodiments of the present application provide a text recognition method and an electronic device. The method comprises: an electronic device can obtain an image of a first text region of a to-be-recognized object and first text content. The electronic device classifies the image of the first text region and the first text content, and displays a text recognition result of the first text region based on a classification result. The classification result comprises a first classification, a second classification or a third classification. The text recognition result corresponding to the first classification is filtered first text content. The text recognition result corresponding to the second classification comprises text content corrected from the first text content. The text recognition result corresponding to the third classification comprises the first text content. In this way, the electronic device can comprehensively consider the image and the text content of the text region to avoid the occurrence of text content with semantic errors in the text recognition result, thereby improving the user experience.

Description

Technical Field

[0001] This application relates to the field of terminal devices, and more particularly to a text recognition method and an electronic device. Background Technology

[0002] With the continuous development of communication technology, mobile phones and other terminals have become an indispensable part of people's daily lives. Users can not only communicate with other users using mobile phones, but also browse or process various types of information.

[0003] During use, for content displayed on the phone that interests the user, such as text in an image or application interface, the user can use the application's text recognition function to identify the text. Typically, text recognition is based on Optical Character Recognition (OCR) technology. Taking images as an example, applications can use OCR technology to recognize text in images and output the recognition results. However, for text recognition scenarios containing truncated text, current OCR technology produces output results that differ significantly from the original text, impacting the user experience. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a text recognition method and an electronic device. In this method, the electronic device can output text recognition results that meet user needs based on the image and text content of a text region.

[0005] Firstly, embodiments of this application provide a text recognition method. The method includes: an electronic device performing text region detection on an object to be recognized to obtain an image of a first text region, wherein the first text region includes text content. The electronic device performs text content recognition on the obtained first text region to obtain first text content. Next, the electronic device classifies the image of the first text region and the first text content to obtain a classification result. Subsequently, the electronic device displays the text recognition result of the first text region based on the classification result. The step of displaying the text recognition result may specifically include: if the classification result is a first category, the text recognition result filters out the first text content. If the classification result is a second category, the text recognition result includes the corrected text content of the first text content. If the classification result is a third category, the text recognition result includes the first text content. In this way, the electronic device can comprehensively consider image information (i.e., the image of the text region) and text information (i.e., the text content), and can filter out the text content recognition result (i.e., the first text content) when a significant amount of text content in the text region is missing. When less text content is missing, the corrected result is output. Furthermore, it can output the corresponding text when no text content is missing. This allows the text recognition results to be presented as correct and semantically coherent, while filtering out semantically incorrect results (i.e., text content), thereby achieving a more human-like and complex decision-making effect and improving the user experience.

[0006] For example, the text recognition result may optionally be Figure 4 The text recognition result display box 405 is used in this context. Specifically, if the text recognition result is the result of the first classification indicator (i.e., filtering), the result corresponding to the first text region in the text recognition result display box 405 will be empty, meaning the text content recognition result (i.e., the first text content) corresponding to the first text region will not be displayed. If the text recognition result is the result of the second classification indicator (i.e., outputting corrected text content) or the result of the third classification indicator (i.e., directly outputting text content), then the text recognition result display box 405 will include either the corrected text content corresponding to the first text region or the text content of the first text region.

[0007] For example, the text recognition result can be the result corresponding to the text region itself. For instance, if the text recognition result is the result of the first classification indication (i.e., filtering), the text recognition result corresponding to the first text region displayed by the electronic device will be empty (it can be blank or there may be no blank). If the text recognition result is the result of the second classification indication (i.e., outputting corrected text content) or the result of the third classification indication (i.e., directly outputting text content), the electronic device can display the text content corresponding to the first text region (which can be modified or the result after text content recognition) in the text recognition result display box 405.

[0008] For example, the classification result may optionally be a numerical value used to represent the classification item.

[0009] For example, the classification result can also include 3 values, and the classification corresponding to the largest value is the classification corresponding to the first text region.

[0010] According to the first aspect, the electronic device classifies the image and the first text content based on the first text region to obtain a classification result, including: the electronic device obtains intermediate representation information based on the image and the first text content of the first text region. The electronic device classifies the intermediate representation information to obtain a classification result. In this way, the electronic device utilizes high-dimensional multimodal semantic information to make more refined decisions on different combinations of inputs, thereby achieving a complex decision-making effect that resembles human behavior.

[0011] For example, intermediate representation information can be referred to as multimodal information.

[0012] For example, intermediate representation information can be used to represent the image features of the first text region and the text features of the first text content.

[0013] According to the first aspect, or any implementation of the first aspect above, the electronic device classifies intermediate representation information to obtain a classification result, including: the electronic device classifies the intermediate representation information using a classification model to obtain a classification result. In this way, the electronic device can classify intermediate representation information using a pre-trained classification model to obtain the corresponding classification result.

[0014] According to the first aspect, or any implementation of the first aspect above, before the electronic device displays the text recognition result of the first text region based on the classification result, the method further includes: the electronic device correcting the intermediate representation information to obtain the corrected text content. For example, the electronic device corrects the intermediate representation information before, simultaneously with, or after classifying the intermediate representation information to obtain the corrected text content. The electronic device may determine whether to output the corrected text content based on the classification result. For example, if the corrected text content is not required to be output, such as if the classification result is a first category or a third category, the corrected text content is discarded.

[0015] According to the first aspect, or any implementation of the first aspect above, the electronic device corrects the intermediate representation information to obtain the corrected target text content, including: the electronic device corrects the intermediate representation information through a correction model to obtain the corrected text content of the first text content. In this way, the electronic device can correct the intermediate representation information through a pre-trained correction model to obtain the corrected text content.

[0016] According to the first aspect, or any implementation thereof, the electronic device obtains intermediate representation information based on the image and the first text content of the first text region, including: the electronic device performing image encoding on the image of the first text region to obtain first image encoding information; the electronic device performing text encoding on the first text content to obtain first text encoding information; and the electronic device performing multimodal encoding on the first image encoding information and the first text encoding information using a multimodal encoding model to obtain intermediate representation information. In this way, the electronic device can obtain higher-dimensional semantic information by encoding the image and text content of the text region. The electronic device can use a pre-trained multimodal encoding model to perform multimodal encoding on the first image encoding information and the first text encoding information to obtain intermediate representation information with high-dimensional semantics.

[0017] According to the first aspect, or any implementation thereof, a multimodal coding model, a classification model, and a correction model constitute a neural network. The training data of the neural network includes a second text region and its corresponding second text content, as well as a third text region and its corresponding third text content. The second text region includes partially missing text content, while the text content in the third text region is complete text content. Thus, by inputting images and text content of different types of text regions (including text regions with and without missing text), the neural network can be trained iteratively to enable it to perform the corresponding function: fusing, classifying, and correcting images and text content of text regions.

[0018] According to the first aspect, or any implementation of the first aspect above, the text recognition result of the first text region is displayed in the text recognition region, which also includes the text content corresponding to the third text region of the object to be recognized. Thus, the text recognition method in this application can implement different processing methods for text content; that is, the final displayed text recognition result is always semantically coherent text content. For semantically incoherent text content in the text recognition result, filtering or correction methods are used to avoid the influence of semantically incoherent text content on the text recognition result.

[0019] According to the first aspect, or any implementation of the first aspect above, if the first text region includes partially missing text content, the text recognition result is either a first category or a second category. For example, the partially missing text content could mean that each character in the text region is missing some information, such as the upper half or the lower half. Alternatively, the partially missing text could also mean that at least one character in the text region is missing some information.

[0020] According to the first aspect, or any implementation of the first aspect above, the semantics expressed by the first text content are different from the semantics expressed by the text content in the first text region. Thus, in this embodiment of the application, the text content recognition results can be filtered to remove or correct text content with different semantics from the original, thereby improving the user experience.

[0021] According to the first aspect, or any of the implementations of the first aspect above, the object to be identified is an image, webpage, or document.

[0022] Secondly, embodiments of this application provide a text recognition method. The method includes: an electronic device performing text region detection on an object to be recognized to obtain an image of a first text region; the first text region includes text content. The electronic device performs text content recognition on the first text region to obtain first text content. The electronic device displays the text recognition result of the first text region based on the image of the first text region and the first text content. The display of the text recognition result of the first text region based on the image of the first text region and the first text content includes: if the image of the first text region indicates that the first text region includes partially missing text content and the first text content is semantically coherent, or if the image of the first text region indicates that the first text region does not include partially missing text content, the text recognition result includes the first text content; if the image of the first text region indicates that the first text region includes partially missing text content and the first text content includes semantically incorrect text content, the text recognition result filters out the first text content or the text recognition result includes corrected text content. Thus, the electronic device can comprehensively consider image information (i.e., the image of the text region) and text information (i.e., the text content), and can filter the text content recognition result (i.e., the first text content) when the text region contains a large amount of missing text content. When minimal text content is missing, the system outputs the corrected result. Conversely, it can output the corresponding text even when no text content is missing. This allows the system to present correct and semantically coherent results in the text recognition output, while filtering out semantically erroneous results (i.e., text content), thereby achieving a more human-like and complex decision-making effect and improving the user experience.

[0023] For example, an electronic device can detect whether the text content in a text region is truncated, i.e., whether it includes missing text, based on an image of the text region. In one example, if the text content is not truncated, the first text content can be directly output. In another example, if the text content is truncated, the semantics of the first text content are checked. If the semantics of the first text content are coherent, the first text content can be directly output. If the semantics of the first text content are incoherent, the first text content is further checked to see if it can be modified. If the first text content can be modified, the modified text content is output; if the first text content cannot be modified, the first text content is filtered.

[0024] According to the second aspect, the electronic device displays the text recognition result of the first text region based on the image of the first text region and the first text content, including: if the image of the first text region represents that the first text region includes partially missing text content, and the first text content includes semantically incoherent text content, the electronic device detects whether the first text content can be corrected. If the first text content cannot be corrected, the text recognition result filters out the first text content. If the first text content can be corrected, the text recognition result includes the corrected text content. Thus, when the electronic device detects that the text content in the first text region is truncated and the first text content is semantically incoherent, it can further detect whether the first text content can be corrected. If it can be corrected, the electronic device can correct the first text content and output the corrected text content. If it cannot be corrected, the electronic device filters out the first text content. In other words, the text recognition result of the first text region displayed by the electronic device is either empty, or contains the corrected text content, or originally semantically coherent text content, to avoid the impact of incorrect text content recognition results on user experience.

[0025] According to the second aspect, or any implementation of the second aspect above, if the first text content can be corrected, the method further includes: the electronic device correcting the first text content using a correction model to obtain corrected text content. In this way, the electronic device can correct the first text content using a pre-trained correction model to obtain semantically coherent text content.

[0026] According to the second aspect, or any implementation thereof, the electronic device displays the text recognition result of the first text region based on the image of the first text region and the first text content, including: the electronic device classifying the image of the first text region using a classification model to obtain a classification result; the classification result is used to indicate whether the first text region includes partially missing text content. In this way, the electronic device can classify the image of the text region using a pre-trained classification model to detect whether the text content in the text region has been truncated.

[0027] According to the second aspect, or any implementation of the second aspect above, if the image representation of the first text region includes partially missing text content, the electronic device displays the text recognition result of the first text region based on the image and the first text content. This includes: the electronic device performing semantic analysis on the first text content using a semantic model to obtain a semantic analysis result; the semantic analysis result is used to indicate whether the first text content includes semantically erroneous text content. In this way, the electronic device can perform semantic analysis on the text content using a pre-trained semantic model to obtain a semantic analysis result.

[0028] For example, the semantic analysis result can be a numerical value, and the electronic device can preset a semantic coherence threshold, which is used to indicate the semantic coherence of the text content. If the numerical value of the semantic analysis result is greater than or equal to the threshold, the first text content is semantically coherent; if the numerical value of the semantic analysis result is less than the threshold, the first text content is semantically incoherent.

[0029] According to the second aspect, or any implementation of the second aspect above, the semantic analysis result is further used to indicate whether the first text content can be corrected. The electronic device displays the text recognition result of the first text region based on the image of the first text region and the first text content, including: the electronic device determining whether the first text content can be modified based on the semantic analysis result. The electronic device can set a correction threshold, which is different from the semantic coherence threshold. If the value of the semantic analysis result is greater than or equal to the correction threshold, the first text content can be corrected. If the value of the semantic analysis result is less than the correction threshold, the first text content cannot be corrected.

[0030] According to the second aspect, or any implementation of the second aspect above, a neural network is composed of a correction model, a classification model, and a semantic model. The training data of the neural network includes a second text region and the corresponding second text content, as well as a third text region and the corresponding third text content. The second text region includes partially missing text content, while the text content in the third text region is complete text content. In this way, by inputting images and text content of text regions of different types (including text regions with missing and non-missing text), the neural network can be trained iteratively to enable the neural network to perform the corresponding functions, namely, to perform truncation judgment, semantic analysis, and correction on images and text content of text regions.

[0031] According to the second aspect, or any implementation of the second aspect above, the text recognition result of the first text region is displayed in the text recognition region, which also includes the text content corresponding to the third text region in the object to be recognized.

[0032] According to the second aspect, or any implementation of the second aspect above, the semantics expressed by the semantically incorrect text content are different from the semantics expressed by the corresponding text content in the first text region.

[0033] According to the second aspect, or any implementation of the second aspect above, the object to be identified is an image, webpage, or document.

[0034] Thirdly, embodiments of this application provide an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when executed by the one or more processors, cause the electronic device to perform instructions of the method in the first aspect or any possible implementation thereof.

[0035] Fourthly, embodiments of this application provide an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when executed by the one or more processors, cause the electronic device to perform instructions of the method in the second aspect or any possible implementation thereof.

[0036] Fifthly, embodiments of this application provide a computer-readable medium for storing a computer program, the computer program including instructions for performing the method in the first aspect or any possible implementation of the first aspect.

[0037] In a sixth aspect, embodiments of this application provide a computer-readable medium for storing a computer program, the computer program including instructions for performing the methods in the second aspect or any possible implementation thereof.

[0038] In a seventh aspect, embodiments of this application provide a computer program including instructions for performing the method in the first aspect or any possible implementation thereof.

[0039] Eighthly, embodiments of this application provide a computer program including instructions for performing the method in the second aspect or any possible implementation thereof. Attached Figure Description

[0040] Figure 1 A schematic diagram of the hardware structure of an electronic device as an example;

[0041] Figure 2 A schematic diagram of the software structure of an electronic device as an example;

[0042] Figure 3 This is an illustrative diagram of a text recognition scenario containing truncated text.

[0043] Figure 4 This is an illustrative diagram illustrating an application scenario of the text recognition method in this application embodiment;

[0044] Figure 5 This is a schematic diagram illustrating a text recognition method as an example.

[0045] Figure 6 A schematic diagram illustrating text recognition as an example;

[0046] Figure 7 This is an example illustration of text image encoding.

[0047] Figure 8 This is an example of an image information encoding diagram;

[0048] Figure 9 This is an example of an image information encoding diagram;

[0049] Figure 10 This is an example of an image patch flattening diagram;

[0050] Figure 11 This is an example illustration of text content encoding.

[0051] Figure 12 This is a schematic diagram illustrating the text information encoding process as an example.

[0052] Figure 13 This is a schematic diagram illustrating the process of obtaining intermediate representation information;

[0053] Figure 14a This is an example of a multimodal coding diagram;

[0054] Figure 14b This is a schematic diagram of the processing flow of a multimodal encoder;

[0055] Figure 14c This is a schematic diagram illustrating the classification process as an example.

[0056] Figure 15 This is an example illustration of text correction.

[0057] Figure 16 This is a schematic diagram illustrating the processing flow of the correction module as an example.

[0058] Figure 17 This is a schematic diagram illustrating the processing flow of a Transformer Decoder as an example.

[0059] Figure 18a This is a schematic diagram illustrating an application scenario;

[0060] Figure 18b This is an illustrative diagram of another application scenario.

[0061] Figure 18c This is an example illustration of another application scenario;

[0062] Figure 18d This is an example illustration of another application scenario;

[0063] Figure 18e This is an example illustration of another application scenario;

[0064] Figure 19 This is a schematic diagram illustrating a text recognition method as an example.

[0065] Figure 20 This is an example illustration of text image processing;

[0066] Figure 21 The processing flow of the semantic model is shown as an example.

[0067] Figure 22 This is a schematic diagram of the structure of an exemplary device. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] Figure 1 A schematic diagram of the structure of the electronic device 100 is shown. It should be understood that... Figure 1 The electronic device 100 shown is merely an example of an electronic device, and the electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. Figure 1 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0070] Electronic device 100 may include: processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0071] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0072] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0073] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0074] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0075] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0076] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0077] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0078] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0079] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0080] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0081] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. Electronic device 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to display screen 194 and application processor. The GPU performs mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0082] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0083] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0084] The ISP is used to process data fed back by the camera 193. The camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0085] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0086] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0087] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0088] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0089] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100. In other embodiments, this application embodiment can also be applied to other systems such as HarmonyOS, and the implementation methods can all refer to the technical solutions in the embodiments of this application; therefore, this application will not provide detailed examples of each.

[0090] Figure 2 This is a software structure block diagram of an electronic device 100 according to an embodiment of this application.

[0091] The layered architecture of the electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0092] The application layer can include a series of application packages.

[0093] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, text recognition, and text processing. The text recognition application in this embodiment may also be referred to as a text recognition module or text recognition engine, etc., and this application is not limited thereto. The text recognition module can be used to identify text regions and text content in the image to be recognized (specific concepts are explained below). The text processing application may also be referred to as a text processing module, used to further process the output results of the text recognition module (specific processing flow is explained in the embodiments below). It should be noted that in this embodiment, the further processing of the results of the text recognition module by the text processing module is used as an example for explanation. In other embodiments, the steps performed by the text processing module may also be executed by the text recognition module, and it can be understood that the steps performed by the text recognition module and the text processing module may be performed by one module, and this application is not limited thereto.

[0094] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0095] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0096] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0097] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0098] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0099] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0100] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0101] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0102] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries, 2D graphics engines (e.g., SGL), etc.

[0103] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0104] The media library supports playback and recording of various common audio and video formats, as well as still image files. It also supports multiple audio and video encoding formats.

[0105] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0106] A 2D graphics engine is a graphics engine for 2D drawing.

[0107] The kernel layer is the layer between hardware and software. It contains at least drivers for display, camera, audio, sensors, Bluetooth, and Wi-Fi.

[0108] Understandable, Figure 2 The components included in the system framework layer, system library, and runtime layer shown do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than shown, or combine some components, or split some components, or have different component arrangements.

[0109] Figure 3 This is an illustrative diagram illustrating a text recognition scenario containing truncated text. Please refer to... Figure 3(1) The image 302 is displayed on the mobile phone's display interface 301. For example, the display interface 301 can be an application interface, such as the interface of a system application like a gallery application, or it can be the application interface of a third-party application like a chat application. That is, in this embodiment, the mobile phone's system can have built-in text recognition functionality (i.e.,...). Figure 2 (The text recognition module in the phone, for example, a gallery application can call the phone's text recognition module to perform text recognition on images. Optionally, third-party applications on the phone can also have built-in text recognition functions. The implementation process of text recognition functions in different third-party applications can be the same or different, and this application does not impose any limitations.)

[0110] Still refer to Figure 3 (1) For example, image 302 includes both text and images (of course, image 302 may also include only text). It should be noted that this application embodiment only uses the text recognition scenario of images as an example for illustration. In other embodiments, it can also be applied to the text recognition scenario in the application interface. For example, the scenario can be the text recognition of the page displayed by the browser application. This application does not limit it.

[0111] Optionally, image 302 may be generated by the mobile phone in response to the user's operation and after performing a screenshot operation; image 302 may also be generated by the mobile phone through the camera function; image 302 may also be a downloaded image, etc., and this application does not limit it.

[0112] For example, the text in image 302 comprises multiple lines, wherein the first and last lines of text displayed in image 302 are truncated by the border of image 302. In this embodiment, such text is referred to as "truncated text". It should be noted that... Figure 3This paper only uses vertically truncated text as an example for illustration. The technical solutions in this application embodiment can also be applied to the recognition scenarios of horizontally truncated text and diagonally truncated text. Specific examples will be described below. For example, the "vertically truncated text" in this application embodiment can optionally be a truncation perpendicular to the direction of the text line. It can be understood that the text line is obscured by the top and bottom edges of the screen or certain fixed or frozen status bars due to the vertical scrolling of the interface. For example, taking image 302 as a screenshot of a webpage, when a user scrolls up and down the webpage, the first line currently displayed on the webpage may be truncated by the top edge of the webpage (which can also be understood as the top border of the display frame). When the user takes a screenshot of the currently displayed webpage, the mobile phone responds to the received screenshot to generate image 302. The first line of text displayed in image 302 is the "vertically truncated text". For example, the "horizontally truncated text" in this application embodiment is a truncation along the direction of the text line, such as when taking a picture or scanning causes the text line to be truncated horizontally. For example, "diagonal phase text" can optionally be a truncation in a direction that forms an angle with the direction of the text line.

[0113] Still refer to Figure 3 (1) Users can long-press image 302. Please refer to [link / reference]. Figure 3 (2) For example, in response to a received long press operation on image 302, the application displays an option box 303. Optionally, the option box 303 may include, but is not limited to, a share option, a favorite option, and a text extraction option 304. The position, size, number, and names of the options included in the option box 303 are merely illustrative examples and are not intended to limit the scope of this application.

[0114] For example, the user clicks the text extraction option 304 to instruct the extraction of text from image 302. In response to the received user action, the mobile phone initiates the text recognition function (as mentioned above, the text recognition function can be the application's built-in text recognition function or it can call the system's text recognition function; this application does not limit this).

[0115] In this embodiment, the text recognition function optionally employs OCR technology. OCR technology mainly consists of two steps: the first step is text region detection, and the second step is text content recognition (also known as text content recognition). For example, the text region detection step may optionally involve detecting at least one text region in the image, i.e., recognizing the region in the image that contains text. For example, the text content recognition step may optionally involve recognizing the text within the acquired text region, i.e., recognizing the specific text content within the text region. Detailed steps for text region detection and text content recognition can be found in existing technical embodiments and will not be repeated here.

[0116] Please refer to Figure 3 (3) of Figure 3 . Exemplarily, the display interface 301 includes, but is not limited to, the scaled-down picture 302 and the text recognition result display box 305. It should be noted that the interface layout in the display interface 301 in the embodiments of the present application is only for illustrative purposes and is not limited in the present application. Exemplarily, the text recognition result display box 305 includes, but is not limited to, the "Smear to Select Text" option, the text recognition result, and other options. Optionally, the other options include, but are not limited to, the "Select All" option, the "Search" option, the "Copy" option, and the "Translate" option, etc. Each option in the other options can be used to perform corresponding processing on the text recognition result.

[0117] Still referring to Figure 3 (3) of Figure 3 . Exemplarily, the text recognition result in the text recognition result display box 305 is the result recognized by the text recognition function. However, in this example, since the first-line text of the picture 302 is truncated, such as the vertically truncated text described above, the first-line text is not completely displayed. Correspondingly, the result recognized by the text recognition function may be inaccurate. For example Figure 3 (3) of Figure 3 shows that the original text of the first-line text in the web page is "In the first round of the game, when Quan ** and others appeared, the whole audience cheered, 5", and due to the first-page text being truncated by the upper border when browsing the web page, the first-line text in the picture 302 after screenshot is truncated. When applying text recognition to the picture 3, after recognizing the first-line text, the output result is "Rikong L loan, Shihong roasted eight yuan's soil from ding, 5", and the difference between it and the original text is relatively large, with semantic and logical errors. And for such recognition results, even through technologies such as semantic reasoning, the original text cannot be restored, affecting the user experience. Exemplarily, the recognition result corresponding to the text line in the picture 302 that is not truncated (such as the second-line text in the picture 302) has no difference from the original text.

[0118] This application provides a text recognition method. This method uses text images and text content as input to a model (which can be called a text recognition model or a text recognition network), obtaining corresponding modal encoding information through their respective modal encodings. A text processing module fuses the modal information of the text image and the text content, using this as attention input to the classification decoder and the correction decoder. In principle, the model implicitly considers both image information (mainly truncation) and text information (mainly semantic coherence), and utilizes high-dimensional multimodal semantic information to make more refined decisions on different input combinations, thus achieving a human-like complex decision-making effect. This complex decision-making, reflected in the final result, results in three classification outcomes: direct filtering indicates cases where occlusion leads to uncorrectable semantics; corrected output indicates cases where occlusion leads to semantic incoherence but is correctable; and no correction, direct output indicates cases where there is no occlusion or occlusion does not affect semantics. In other words, the text recognition method provided in this application embodiment offers a more human-like processing solution. Under normal circumstances, if the text is heavily obscured, the user's naked eye may not be able to recognize the correct information, and the user can also determine that the text content they read from the truncated text is incorrect. If the text is only slightly obscured, the user can semantically determine the obscured text. For unobscured text, the user can correctly read the corresponding text content. The technical solution in this application embodiment achieves a human-like user reading effect, and can output no result when the text is heavily obscured (i.e., truncated), while outputting a corrected result when the obscuration is minimal. Furthermore, it can output the corresponding text when there is no obscuration. Thus, it can present correct and semantically coherent results in the text recognition results, while filtering out semantically incorrect results (i.e., text content), thereby improving the user experience.

[0119] Figure 4 For an illustrative illustration of an application scenario of the text recognition method in this application embodiment, please refer to... Figure 4 (1) Taking a gallery application as an example, after a user clicks on the thumbnail corresponding to image 402 displayed in the gallery application, the gallery application can display image 402 in the display interface 401. Optionally, the display interface 401 may also include, but is not limited to, options (or controls) such as sharing options and favorites options.

[0120] For example, the gallery application can invoke the system's text recognition module and text processing module to perform text recognition and processing on image 402 (also referred to as the image to be recognized or the image to be recognized). As described above, in this embodiment, text recognition includes two parts: text region detection and text content recognition. Optionally, the text recognition module can perform the text region detection step after receiving an operation from the user clicking the thumbnail corresponding to image 402 to detect whether the image 402 includes a text region. In this example, image 402 includes both images and text (of course, it can also include only text, which is not limited in this application). Accordingly, the text recognition module can detect at least one text region included in image 402. After the text recognition module detects that image 402 includes a text region, it can display the "Extract Text from Image" option 403 on the display interface 401. The user can click the "Extract Text from Image" option 403 to instruct the extraction of text content from image 402. In response to the received user operation, the gallery application performs text recognition on image 402 through the text recognition module, that is, performs the text content recognition step to obtain the corresponding text content in each text region. In this embodiment, the text processing module can further process the recognition results (including text regions and text content) obtained by the text recognition module. Please refer to... Figure 4 (2) The display interface 401 includes, but is not limited to, a scaled-down image 402 and a text extraction display box 404. Optionally, the text extraction display box 404 includes, but is not limited to, a text recognition result display box 405 and other options. Other options include, but are not limited to, "smudge selected text" option, "read the full text" option, "select all" option, "search" option, "copy" option, and "translate" option, etc. It should be noted that the layout of each control in the display interface shown in this application embodiment is only an illustrative example and is not limited in this application. For example, the text recognition result display box 405 includes the text content recognized by the text recognition module, such as Figure 4 As shown in (2), in this embodiment of the application, for truncated text (e.g., the first line of text), the mobile phone does not display the corresponding text in the text recognition result display box 405. That is to say, for text recognition results that may contain semantic errors or garbled characters, the text processing module can adopt the method of not outputting (i.e., not displaying) to avoid the problem that the text recognition result is significantly different from the original text. Still referring to Figure 4(2) For untruncated text, the text processing module can display the corresponding text in the text recognition result display box 405. Optionally, in this embodiment, the text processing module can also correct (or revise) the text content recognized by the text recognition module to obtain correct text (which can also be understood as text that is close to or the same as the original text), and output (i.e., display in the text recognition result display box 405) the corrected result. That is to say, in this embodiment, by filtering or correcting text with semantic errors, the text recognition result displayed in the text recognition result display box 405 is semantically logically correct and coherent, thereby improving the user experience.

[0121] It should be noted that the embodiments of this application only use the text recognition and processing scenario of images as an example for illustration. In other embodiments, it can also be applied to the text recognition and processing scenario in the application interface. For example, the scenario can be the text recognition and processing of the page displayed by the browser application. This application does not limit it.

[0122] It should be further noted that image 402 may be generated by the mobile phone in response to the user's operation and after performing a screenshot operation; image 402 may also be generated by the mobile phone through the camera function; image 402 may also be a downloaded image, etc., and this application does not limit it.

[0123] It should be further noted that the embodiments in this application only illustrate the scenario of a gallery application calling the text recognition module and the text processing module. The steps performed by the text recognition module and the text processing module in the embodiments of this application can also be applied to other applications. For example, the text recognition function built into a chat application can perform text recognition on the image to be recognized and obtain the corresponding text recognition result. The chat application can call the mobile phone's text processing module to further process the text recognition result. Alternatively, the chat application can also have its own text recognition module and text processing module, and implement the steps implemented by the text recognition module and text processing module involved in the embodiments of this application. Furthermore, the chat application can also call the mobile phone's text recognition module and text processing module; this application does not impose any limitations.

[0124] It should be further noted that the steps performed by the text recognition module in this application embodiment are merely illustrative examples. The steps performed by the text recognition module in the mobile phone and the text recognition module built into the application may be the same or different. For specific details, please refer to existing technical embodiments, which are not limited in this application. For example, the text recognition module in the mobile phone can use OCR technology to perform text recognition and obtain the corresponding recognition results, including text images and text content (the concepts of text images and text content will be explained below). The text recognition module in the chat application can use other technologies to perform text recognition and obtain the corresponding recognition results, which also include text images and text content. Optionally, the recognition results obtained by the text recognition module of the chat application and the text recognition module of the mobile phone application may be the same or different. For example, the text recognition module in the mobile phone may recognize 5 text regions and obtain the corresponding text content. The text recognition module in the chat application may recognize 6 text regions and obtain the corresponding text content. This application is not limited in this respect. That is to say, the text processing module in this application embodiment can further process the recognition results of any text recognition module (which may be from the mobile phone and / or the application) to obtain results that meet the user's needs.

[0125] It should be further noted that the operations that trigger text recognition and processing functions may be the same or different for different applications. The user operation involved in this application (i.e., clicking the "Extract Text" option) is only an illustrative example and is not intended to limit the scope of this application.

[0126] It should be further noted that the embodiments of this application only use the scenario of first-line text truncation as an example for illustration. In other embodiments, the text recognition method in the embodiments of this application can also be applied to scenarios including last-line text truncation.

[0127] It should be further noted that the text is truncated by a border in this embodiment. In other embodiments, the text may be truncated by an image or other reasons, which are not limited in this application.

[0128] In one possible implementation, the text recognition module can perform text region detection on each image in the gallery application while the phone is in standby mode or the gallery application is in the background. In other words, the text recognition module can pre-process the images in the gallery application to perform text region detection, so that when the user clicks on an image containing text regions, a "Extract text from image" option box can be displayed immediately, thereby improving the overall efficiency of text recognition and processing.

[0129] The text recognition method in the embodiments of this application will be described in detail below with reference to the accompanying drawings. Figure 5 This is a flowchart illustrating an exemplary text recognition method. Please refer to... Figure 5 The text recognition module can obtain the results recognized based on OCR technology. The results include at least one text image and the text content corresponding to each text image. For example, Figure 6 For an illustrative diagram of text recognition, please refer to... Figure 6 The text recognition module uses OCR technology to perform text region detection on image 601 (i.e., image 402, for details please refer to image 402, which will not be repeated here) to obtain at least one text region. Specifically, text region detection can be understood as the OCR technology detecting regions containing text in image 601, then segmenting at least one text region in image 601 to obtain at least one text image (i.e., the image corresponding to at least one text region in image 601). For example, as... Figure 6 As shown, the text recognition module detects a text region 602a containing text in image 601. The text recognition module can segment the text region 602a (for example, segment along the dotted line) to obtain the image corresponding to the text region 602a, which is simply referred to as text image 602a.

[0130] For example, the text recognition module can sequentially segment the regions containing text in image 601, such as obtaining an image of text region 603a, referred to simply as text image 603a. This embodiment only uses text regions 602a and 603a as examples; the text recognition module can obtain more text regions in image 601.

[0131] In one possible implementation, after the text recognition module recognizes the text region using OCR technology, it can undergo processing such as radial or perspective transformation correction to obtain the corresponding text image.

[0132] In another possible implementation, the size of a single text image can be the same as or larger than the actual area occupied by the text content within the text image. For example, text image 602a has a size larger than the actual area occupied by the text content; that is, there is a blank area between the border of the text image and the text content (i.e., the edge of the text content).

[0133] Still refer to Figure 6, the text recognition module can recognize the text content of at least one text area (i.e., text image) obtained through OCR technology. Still taking the text image 602a and the text image 603a as examples, the text recognition module recognizes the text content of the text image 602a and obtains the text content recognition result 602b (which can also be called the text content 602b), that is, it is recognized that the text content in the text image 602a is "日孔L贷,士红烤守八元的土从 叮,5". The text recognition module continues to recognize other text images to obtain the corresponding text content recognition results. For example, the text recognition module recognizes the text content of the text image 603a through OCR technology to obtain the corresponding text content recognition result 603b (which can also be called the text content 603b), that is, it is recognized that the text content in the text image 603b is "位冠军也展示了高超的实力,第一轮107B". It should be noted that in this embodiment, only the text images 602a and 603a are used as examples for illustration. The text recognition module can recognize the text content of each obtained text image based on OCR technology to obtain the corresponding text content, and this application will not explain them one by one. Further, it should be noted that the text recognition module can perform text content recognition on each text image in parallel or sequentially, and this application does not make a limitation.

[0134] Still referring to Figure 5 , exemplarily, the text processing module obtains the recognition results obtained by the text recognition module, such as including but not limited to: the text image 602a and the corresponding text content 602b, and, the text image 603a and the corresponding text content 603c. The text processing module executes the Figure 5 process for each text image input by the text recognition module and the text content corresponding to the text image. It should be noted that after the text recognition module obtains the images corresponding to all text areas of the image to be recognized (such as the picture 601) and the corresponding text content, it can output the recognition results to the text processing module for further processing. The text recognition module can execute the Figure 5 process for each obtained text image and text content one by one. The text recognition module can also process multiple text images and text contents in parallel, and this application does not make a limitation. Optionally, the text recognition module can also output the text content and the corresponding text image to the text processing module for processing after obtaining one text content, and this application does not make a limitation and will not be repeated hereinafter.

[0135] Please continue to refer to Figure 5For example, taking text image 602a and text content 602b as examples, the text processing module uses an encoding model (also called an encoding module) to obtain image encoding information corresponding to text image 602a and text encoding information corresponding to text content 602b. Optionally, the encoding model may include, but is not limited to, an image encoding model (also called an image encoding module) and a text encoding model (also called a text encoding module). For example, the image encoding model can be used to encode text image 602a to obtain image encoding information corresponding to text image 602a. That is, the image encoding model can encode text image into semantic information that can be recognized or understood by a machine. For example, the text encoding module can be used to encode text content 602b to obtain text encoding information. It can also be understood that the text encoding module encodes text content into semantic information that can be recognized or understood by a machine.

[0136] It should be noted that the structure of image encoding information and text encoding information can adopt the corresponding encoder architecture according to the encoding process. The encoder described in the embodiments of this application is only an illustrative example and can be set according to actual needs. This application does not limit it.

[0137] It should be further noted that the text processing module can process the text image 602a and the text content 602b sequentially or in parallel, and this application does not impose any limitations. For example, the text processing module can first process the text image 602a to obtain image encoding information, and then process the text content 602b to obtain text encoding information. As another example, the text processing module can first encode the text content 602b, and then encode the text image 602a. Yet another example is that the text processing module can encode both the text image 602a and the text content 602b simultaneously, and this application does not impose any limitations.

[0138] Still refer to Figure 5 For example, taking text image 602a and text content 602b as examples, the text processing module fuses the image encoding information corresponding to text image 602a and the text encoding information corresponding to text content 602b through a multimodal model (which can also be called a multimodal encoding module, multimodal fusion module, etc., which are not limited in this application) to obtain multimodal encoding information, which can also be called intermediate representation information.

[0139] For example, the text processing module corrects the intermediate representation information using a correction model (also called a correction module), and then classifies the intermediate representation information using a classification model (also called a classification module) to obtain a classification result. In this embodiment, the classification result includes three categories: filtering, correction and output, and direct output. The filtering classification item can optionally be filtering the text content, i.e., not displaying the corresponding text content in the text recognition result. The correction and output classification item can optionally be outputting the corrected text, which can also be understood as the text content being corrected before being displayed in the text recognition result. The direct output classification item can optionally be displaying the text content in the text recognition result. In other words, the text processing module can directly display the text content recognized by the text recognition module using OCR technology in the text recognition result. Taking the intermediate representation corresponding to text image 602a and text content 602b as an example, in one example, if the classification result of the intermediate representation information is a filtering classification item, then the text processing module will filter the text content 602b, i.e., not display the text content 602b in the text recognition result, to avoid semantically incorrect text affecting the text recognition result. In another example, if the classification result of the intermediate representation information is a corrected output classification item, the text processing module can display the corrected result of the intermediate representation information in the text recognition result. In yet another example, if the classification result of the intermediate representation information is a direct output, the text processing module displays the text content 602b in the text recognition result.

[0140] The following uses text image 602a and text content 602b as examples to illustrate... Figure 5 The various processes are explained in detail below. Figure 7 is an exemplary diagram illustrating text image encoding. Please refer to... Figure 7 In this embodiment of the application, the text processing module (specifically, the image encoding model) performs image information encoding on the text image 602a, including Patch Embedding and Positional Encoding, thereby converting the three-dimensional image information into two-dimensional image encoded information E. v .

[0141] It should be noted that, as mentioned above, the structure of the encoded information (e.g., two-dimensional encoded information) obtained from encoding text images and text content is based on the encoder architecture. The encoder architecture can be set according to actual needs. For example, in other embodiments, three-dimensional image information can also be converted into higher-dimensional or lower-dimensional image encoded information. This application does not limit this, and it will not be repeated below.

[0142] It should be further noted that the embodiments of this application only use the encoding method of Patch Embedding and PositionalEncoding to encode text images as an example. In other embodiments, other encoding methods can also be used, and this application does not limit them.

[0143] The specific processes of Patch Embedding and Positional Encoding include, but are not limited to:

[0144] (1) The text processing module divides the text image 602a into N patches.

[0145] Figure 8 This is an exemplary diagram illustrating image information encoding. Please refer to... Figure 8 Optionally, the text processing module (specifically, an image encoding model, which will not be repeated below) can resize the height (or width, or both) of the text image 602a to adjust its height to a preset pixel value. For example, the text module can adjust the height of the text image 602a to 32 pixels (or 64 pixels, which can be set according to actual needs; this application does not limit this). Correspondingly, the width of the text image 602a is adjusted proportionally (i.e., the aspect ratio of image 602a) along with its height. Figure 8 As shown, in this embodiment, the adjusted height of text image 602a is H, and the width (or length) is W, which is used as an example for illustration. It should be noted that in other embodiments, the text image may not be resized, and this application does not limit this.

[0146] Still refer to Figure 8 For example, the text processing module divides the text image 602a into N ImagePatches. In this embodiment, assuming the width of an Image Patch is w and the height is h, the number of Image Patches obtained by the text processing module is:

[0147] N=L*W / h*w (1)

[0148] Optionally, the values ​​of h and w can be the same or different, for example, both can be 16 pixels. This can be set according to actual needs, and this application does not limit it.

[0149] Optionally, N can be a positive integer, for example, N can be obtained by rounding up.

[0150] (2) The text processing module performs patch embedding on N image patches.

[0151] Figure 9 This is a schematic diagram illustrating image information encoding as an example. Please refer to... Figure 9 For example, the PatchEmbedding process includes, but is not limited to, the following steps:

[0152] Step a. The text processing module flattens each Image Patch to obtain a one-dimensional vector P corresponding to each Image Patch. i .

[0153] Specifically, each Image Patch has a width of w, a height of h, and the number of channels of c. Correspondingly, the size of each Image Patch is (h*w*c). The text processing module flattens the Image Patch, obtaining a one-dimensional vector of length (h*w*c). For the i-th image patch, this one-dimensional vector is denoted as P. i P i Represented as:

[0154]

[0155] For example, Figure 10 This is an example illustration of an image patch flattening diagram. Please refer to... Figure 10 ,by Figure 8 Taking Image Patch801 as an example, the size of Image Patch801 is (h*w*c). After the text processing module expands ImagePatch801, it obtains the corresponding one-dimensional vector P1, which is represented as:

[0156]

[0157] This is a one-dimensional vector of length (h*w*c). The text processing module can flatten each ImagePatch using the above method to obtain N Pi, as shown below. Figure 9 P1……P shown n .

[0158] Step b. The text processing module passes N one-dimensional vectors Pi through a fully connected layer to obtain N one-dimensional tensors of a preset length.

[0159] For example, refer to Figure 9 The text processing module passes N one-dimensional vectors Pi through fully connected layers with an output length of embedding_size (which can be set according to actual needs, and is not limited in this application) to obtain N one-dimensional tensors E with a length of embedding_size. vi E vi Represented as:

[0160]

[0161] For example, such as Figure 9 As shown, the text processing module passes P1 through a fully connected layer of length embedding_size to obtain a one-dimensional tensor E of length embedding_size. v1 E v1 Represented as:

[0162]

[0163] The text processing module performs the same processing on N one-dimensional tensors in the above manner to obtain E. v1 ...E vn .

[0164] It should be noted that the embodiments of this application only use the preset length of embedding_size as an example for illustration. In other embodiments, the preset length can be other values, which are related to the type of fully connected layer used, and this application does not limit it.

[0165] Step c, the text processing module converts N one-dimensional tensors E vi Arrange them in order to obtain a two-dimensional tensor with dimension N*embedding_size.

[0166] For example, refer to Figure 9 The text processing module will process N one-dimensional tensors E v1 ...E vn Arranged in order, we obtain the two-dimensional tensor E. v0 E v0 Represented as:

[0167]

[0168] Among them, E v0 The dimension is (N*embedding_size).

[0169] It should be noted that the image encoding method in the embodiments of this application is only an illustrative example. For example, in other embodiments, the text processing module can also obtain the image encoding by calling a convolutional kernel with a kernel size of (h*w), a stride of h (or w), and an output channel number of embedding_size, and applying it to the image patches. The specific method can be set according to actual needs. The purpose is to encode N image patches and obtain machine-encoded information with higher semantics.

[0170] Optionally, in embodiments of this application, the text processing module can convert E v0 With classification head E cls By concatenating the components, we obtain the two-dimensional tensor E. v1 Optionally, E cls The dimension can optionally be (1, embedding_size), which can be set according to actual needs and is not limited in this application. Optionally, the classification header E cls These are the learnable parameters of the neural network.

[0171] For example, E v1 It can be represented as:

[0172] E v1 =[E cls E v0 (2)

[0173] For example, suppose the classification head E cls Represented as:

[0174]

[0175] E in the above embodiments v0 For example, the text processing module will use E v0 With E cls By splicing, we get E. v1 E v1 Represented as:

[0176]

[0177] Among them, E v1 The dimension is (N+1, embedding_size).

[0178] It should be noted that in the embodiments of this application, only E is used. v0 With E cls The example given is splicing. In other embodiments, it can also be addition, fusion, or other methods. This application does not limit the methods.

[0179] (3) Text processing module for E v1 Perform Positional Encoding.

[0180] For example, the text processing module will process the two-dimensional tensor E obtained above. v1 With two-dimensional position code E pos Add them together to obtain the image encoding information E. v For example, image encoding information E v It can be represented as:

[0181] E v =Ev1 +E pos (3)

[0182] It should be noted that the dimension of the position encoding is related to the dimension of the result after processing above. This application only uses two dimensions as an example for illustration, and this application does not limit it.

[0183] For example, suppose E pos Represented as:

[0184]

[0185] Among them, E pos The dimension is (N+1, embedding_size). Optionally, E pos Let N be the learnable parameters of the neural network. In this embodiment, for ease of representation, let N be denoted as N. v =N+1.

[0186] like Figure 9 As shown in the above text, E v1 For example, correspondingly, E v1 Image encoding information E is obtained through Positional Encoding. v Represented as:

[0187]

[0188] It should be noted that the embodiments of this application only use the addition method to combine image encoding and position encoding as an example. Other combinations are possible in other embodiments, and this application does not limit them.

[0189] Figure 11 This is an illustrative diagram of text content encoding. Please refer to... Figure 11 In this embodiment of the application, the text processing module (specifically, the text encoding model, which will not be described again below) performs text information encoding (also known as text information encoding) on ​​the text content 602b, including Word Embedding and Positional Encoding, thereby converting the text information into text-encoded information (also known as text-encoded information) with higher semantic features, denoted as E. t .

[0190] It should be noted that the embodiments of this application only use the encoding method of Word Embedding and PositionalEncoding to encode text information as an example. In other embodiments, other encoding methods can also be used, and this application does not limit them.

[0191] Figure 12 For an exemplary schematic diagram of the text information encoding process, please refer to Figure 12 , and the process includes but is not limited to the following steps:

[0192] (1) The text processing module performs word segmentation on the text content 602b.

[0193] For example Figure 12 As shown,示例性的, the text processing module segments the text content 602b according to a preset character length to obtain a word segmentation result (which can also be called a word segmentation sequence).

[0194] In the embodiment of the present application, taking the preset character length as one character as an example, that is, the text processing module divides each character (including punctuation marks) into a word to obtain m words (for example, m is 18, that is, divided into 18 words), that is, a word segmentation sequence w with a sequence length of m, and w can be expressed as:

[0195] w = [w1, w2,... w m

[0196] It should be noted that in other embodiments, the preset character length can also be set according to actual needs. For example, it can be two characters, and the present application does not make any limitations. Optionally, the preset character length can also be of unequal length. For example, "目形" can be divided into one word, and "山" can be divided into one word, and the present application does not make any limitations.

[0197] (2) The text processing module obtains the text sequence number sequence corresponding to the word segmentation sequence.

[0198] In the embodiment of the present application, the text processing module can preset a text sequence number table (which can also be called text sequence number information, character code table, etc., and the present application does not make any limitations). The text sequence number table is used to indicate the correspondence between characters (words or characters) and sequence numbers. For example, the sequence number corresponding to "目" in the text sequence number table is "12". For another example, the sequence number corresponding to "关系" in the text sequence number table is "52", and the correspondence between characters and sequence numbers can be set according to actual needs, and the present application does not make any limitations. It should be noted that the correspondence between text and sequence numbers can be saved in the form of a table, or in other ways, and the present application does not make any limitations.

[0199] Optionally, the characters included in the text sequence number table can cover the dictionary, or any book in a professional field, etc., and the present application does not make any limitations.

[0200] For example Figure 12 As shown,示例性的, the text processing module can, based on the text sequence number table, find the sequence number (which can also be called the text sequence number) corresponding to each word segmentation (character or word) in the word segmentation sequence w to obtain the text sequence number sequence n, and n can be expressed as: ​

[0201] n = [n1, n2, ..., n] m ]

[0202] (3) The text processing module obtains a two-dimensional tensor E from the text sequence number n through word embedding. t0 .

[0203] For example, the text processing module can obtain a two-dimensional tensor E by passing the text sequence number n through an embedding layer. t0 E t0 It can be represented as:

[0204] E t0 =Embedding(n) (4)

[0205] For example, in the embodiments of this application, the two-dimensional tensor E t0 It can be represented as:

[0206]

[0207] Among them, the two-dimensional tensor E t0 The dimension is (m, embedding_size).

[0208] It should be noted that E t0 The dimension is related to the embedding layer, and this application does not limit it.

[0209] (4) The text processing module will E t0 Adding it to the position code yields the text information code E. t .

[0210] For example, such as Figure 12 As shown, the text processing module will use E t0 By using the location code E pos Adding them together yields the text information encoding E. t For example, Winn information encoding E t It can be represented as:

[0211] E t =E t0 +E pos ′ (5)

[0212] It should be noted that the dimension of the position encoding is related to the dimension of the result after processing above. This application only uses two dimensions as an example for illustration, and this application does not limit it.

[0213] For example, suppose E pos ' is represented as:

[0214]

[0215] Among them, E pos The dimension of ′ is (m, embedding_size). Optionally, E pos ′ represents the learnable parameters of the neural network. In this embodiment, for ease of representation, N is denoted as N. t =m.

[0216] like Figure 12 As shown in the above text, E t0 For example, the text processing module will use E t0 With E pos 'Add them together to get the text information encoding E' t E t Represented as:

[0217]

[0218] It should be noted that the embodiments of this application only use the addition method to combine text encoding and position encoding as an example. Other combinations are possible in other embodiments, and this application does not limit them.

[0219] For example, the position encoding in the embodiments of this application can be a parameter-learnable embedding layer similar to Bert Positional Embedding, or it can be positional encoding based on sine / cosine transformation similar to the native Transformer architecture. It can be set according to actual needs, and this application does not limit it.

[0220] Still refer to Figure 5 For example, after the text processing module obtains the image encoding information and the text encoding information, it can obtain intermediate representation information based on the image encoding information and the text encoding information. Figure 13 For an illustrative flowchart of the process for obtaining intermediate representation information, please refer to... Figure 13 Specifically, including but not limited to the following steps:

[0221] (1) The text processing module will encode the image information E v With text encoding information E t Feature fusion is performed to obtain the hybrid semantic code E. m (It can also be called mixed encoded information, but this application does not limit it).

[0222] For example, the text processing module (specifically, a multimodal coding model, which will not be described again below) encodes the image information E. v With text encoding information E t By concatenating the components, we obtain the hybrid semantic code E. mFor example, it can be represented as:

[0223] E m =[E v E t (6)

[0224] For example, combining the image encoding information E mentioned above v With text encoding information E t Hybrid semantic encoding E m It can be represented as:

[0225]

[0226] Among them, the hybrid semantic encoding E m The dimension is (N) v +N t (embedding_size)

[0227] It should be noted that in this embodiment, only image encoding information E is used. v With text encoding information E t The fusion method is illustrated by splicing, but other methods can be used in other embodiments, such as addition, etc., which are not limited in this application.

[0228] (2) The text processing module will use mixed semantic encoding E m Multimodal encoders are used to obtain multimodal encoded information (i.e., intermediate representation information).

[0229] Figure 14a For an illustrative example of multimodal coding, please refer to... Figure 14a The text processing module will mix semantic encoding E m The multimodal encoder 1301 obtains multimodal encoded information (i.e., intermediate representation information), denoted as E. IR For example, a multimodal encoder can also be understood as an input multimodal encoded information used to extract high-dimensional semantic information that integrates image and text information.

[0230] Optionally, the multimodal encoder 1301 consists of stacked Transformer Encoders, for example, L stacks. Each Transformer Encoder mainly consists of a multi-head attention layer and layer normalization (i.e., ... Figure 14a Norm and feedforward neural network (i.e.) Figure 14aIt consists of Feed Forward.

[0231] Figure 14b This is a schematic diagram of the processing flow of the multimodal encoder 1301. Please refer to it. Figure 14b In this embodiment, a stacking quantity L of 3 is used as an example for illustration. That is, the multimodal encoder 1301 includes multimodal encoder 1301a, multimodal encoder 1301b, and multimodal encoder 1301c. It should be noted that the number of encoders described in this embodiment is only an illustrative example and can be set according to actual needs. This application does not limit the number of encoders. For example, the hybrid semantic code E m The multimodal encoder 1301a outputs the signal. The output of multimodal encoder 1301a is then used as input to multimodal encoder 1301b for further encoding. Multimodal encoder 1301b encodes the signal based on the output of multimodal encoder 1301a, producing an output that is then used as input to multimodal encoder 1301c. Multimodal encoder 1301c encodes the signal based on the output of multimodal encoder 1301b, producing the multimodal encoded information E. IR E IR It can be represented as:

[0232] E IR =TE(TE(TE(E) m ))) (7)

[0233] TE identifies a single multimodal encoder in multimodal encoder 1301. Multimodal coding information E IR The dimension is (N) v +N t (embedding_size). For example, it can be represented as:

[0234]

[0235] It should be noted that the internal processing flow of each layer in the multimodal encoder 1301 can be referred to the relevant content in the existing technical embodiments, and will not be repeated in this application.

[0236] It should be further noted that the embodiments in this application only use TransformerEncoder as an example of a multimodal encoder. In other embodiments, the multimodal encoder can also be a similar bidirectional recurrent neural network, or a simpler convolutional neural network encoder, which can be set according to actual needs, and this application does not limit it.

[0237] It should be further noted that the method by which the text processing module obtains multimodal encoding is not limited to concatenating image encoding information and text encoding information and then passing them through a multimodal encoder. In other embodiments, the text processing module can also pass the image encoding information and text encoding information through their respective encoders separately and then fuse them. For example, the text processing module can obtain high-dimensional image semantic information by passing the image encoding information through an image encoder and obtain high-dimensional text semantic information by passing the text encoding information through a text encoder. The text processing module then concatenates the high-dimensional image semantic information and high-dimensional text semantic information after dimensional alignment to obtain intermediate representation information. The specific method can be set according to actual needs, with the aim of obtaining high-dimensional image semantic features and text semantic features.

[0238] Please continue to refer to Figure 5 For example, the text processing module (which can be a classification model, as will not be repeated below) can classify intermediate representation information to determine whether to output text content 602b based on the classification results.

[0239] Figure 14c This is a schematic diagram illustrating the classification process. Please refer to... Figure 14c For example, the text processing module can pass multimodal encoded information (i.e., intermediate representation information) through a classification model to obtain a classification result. For example, the classification model can include, but is not limited to, a classification decoder and an argmax layer (or a softmax layer). In this embodiment, the classification decoder is a fully connected layer, and the fully connected layer is an MLP (Multi-layer Perceptron) as an example. For example, the MLP can include multiple hidden layers. It should be noted that in this embodiment, only a fully connected layer (e.g., an MLP) is used as the classification decoder. In other embodiments, the classification decoder can be other decoders, such as, but not limited to, decoders like Transformer Decoder or Recurrent Neural Network (RNN) Decoder, which can be set according to actual needs, and this application does not limit it. Its purpose is to output the corresponding classification result based on the input intermediate representation information. It should be further noted that in this embodiment, only the argmax layer is used as an example; in other embodiments, it can also be an argmax layer and a softmax layer, which can be set according to actual needs, and this application does not limit it. Its purpose is to output the category corresponding to the maximum score.

[0240] Optionally, in this embodiment of the application, the classification result includes, but is not limited to, three classification items:

[0241] (a) Filtering

[0242] (b) Correct and output

[0243] (c) Direct output

[0244] After the multimodal encoded information is passed through the classification decoder, the classification result includes scores for the three classification items. The module can then pass these scores through either an argmax layer or a softmax layer to obtain the final decision category.

[0245] For example, as mentioned above, multimodal coded information E IR The dimension is (N) v +N t (embedding_size). Optionally, in this embodiment, multimodal coding information E can be taken. IR The first dimension yields a one-dimensional tensor E of length embedding_size. IR0 , is represented as:

[0246]

[0247] The text processing module will convert the one-dimensional tensor E IR0 The fully connected layer outputs a one-dimensional tensor T of length 3 (the same as the number of classification items). out Optionally, the fully connected layer can be an MLP, which can include multiple hidden layers. Correspondingly, T out It can be represented as:

[0248] T out =MLP(E IR0 (8)

[0249] Among them, T out The dimension is 3, which can be understood as T. out Including the scores corresponding to the three categories a, b, and c mentioned above, for example, it can be represented as:

[0250] T out = [f(a), f(b), f(c)]

[0251] Where f(a) is the score corresponding to category a (i.e., the filtered category), f(b) is the score corresponding to category b (i.e., the corrected output category), and f(c) is the score corresponding to category c (i.e., the directly output category).

[0252] For example, the text processing module will T outThe argmax layer outputs the classification item corresponding to the maximum score. It should be noted that this embodiment only uses an MLP as the fully connected layer for illustration. In other embodiments, the fully connected layer can be other decoders, such as, but not limited to, decoders like Transformer Decoders or Recurrent Neural Network (RNN) Decoders, which can be set according to actual needs; this application does not impose any limitations. Its purpose is to output the corresponding classification result based on the intermediate representation information of the input. Furthermore, it should be noted that this embodiment only uses the argmax layer as an example; in other embodiments, it can also be an argmax layer and a softmax layer, which can be set according to actual needs; this application does not impose any limitations. Its purpose is to output the classification item corresponding to the maximum score.

[0253] In one example, if f(a) is the maximum value, the output result is 'a', meaning the classification result is the filtered category. Correspondingly, the text processing module can filter the corresponding text content, meaning it won't display the corresponding text content in the text recognition result. For example, during the processing of text image 602a and text content 602b, if the text processing module detects a classification result of class 'a' (i.e., a filtered category), then the text processing module will filter the text content 602b. Figure 4 As shown in (2), the text recognition results do not include the truncated first line of text, thus avoiding errors in the truncated text recognition results and affecting the user experience.

[0254] In another example, if f(c) is the maximum value, the output result is c, meaning the classification result is a direct output of the classification item. In other words, the OCR technology's recognition result is correct. Correspondingly, the text processing module can display the corresponding text content in the text recognition result. For example, during the processing of text image 603a and text content 603b, the text processing module detects a classification result of class c, meaning the classification result is a direct output of the classification item. The text processing module determines that text content 603b can be directly output, such as... Figure 4 As shown in (2), the text processing module can display the text content 603b at the corresponding position in the text recognition result.

[0255] In another example, if f(b) is the maximum value, then the output result is b, meaning the classification result is the corrected output classification item. This can be understood as the OCR technology recognizing results containing some errors, which need to be corrected before output. As mentioned above (i.e.... Figure 5In the text processing module, each piece of multimodal coding information (i.e., intermediate representation information) acquired is corrected by the correction module. Once the text processing module detects that the classification result corresponding to a single piece of multimodal coding information is the corrected output classification item, it can display the corrected text content in the text recognition results. It should be noted that if the classification result is class A or class C, the text processing module discards (or ignores) the corrected result output by the correction module.

[0256] Figure 15 This is an illustrative diagram illustrating text correction. Please refer to... Figure 15 In this embodiment, the correction module includes a Transformer Decoder as an example. The text processing module passes the multimodal encoded information (i.e., intermediate representation information) through a Transformer Decoder 1501, a fully connected layer, and an argmax layer to obtain the corrected text content.

[0257] It should be noted that in other embodiments, the correction module can also be other architectures, such as including but not limited to: a forward decoder based on a recurrent neural network, a Bert Decoder architecture, a decoder similar to stepwise monotonic attention, etc., which can be set according to actual needs, and this application does not limit it. The purpose is to correct the intermediate representation information of the input to obtain the corrected text.

[0258] Please refer to Figure 15 The Transformer Decoder1501 consists of Q stacked Transformer Decoders, where Q can be positive integers greater than 0. A single Transformer Decoder can be represented as a TD, and a single TD includes, but is not limited to: Masked multi-head attention layers, multi-head attention layers, and layer normalization (i.e., ...). Figure 15 (Norm) and feedforward neural networks (i.e.) Figure 15 (Feed forward in the middle). The specific processing details of each layer can be found in the relevant content of existing technical embodiments, and will not be repeated in this application.

[0259] Optionally, in the Transformer Decoder architecture, the K and V vectors of the Transformer Decoder are multimodal encoded information (i.e., the output of the Encoder), and the Q vector is the output of the Masked multi-head attention layer.

[0260] Figure 16Schematic diagram of the processing flow of the correction module shown exemplarily. Please refer to Figure 16 , exemplarily, it is assumed that the recognition result of the OCR technology obtained by the text processing module includes text content and text images, where the text content is "volcanic eruption". That is, the character "暴" is misrecognized. The text processing module obtains multi-modal encoding information corresponding to the text content and text images based on the method in the above embodiment. And, the text processing module obtains the corresponding classification result based on the multi-modal encoding information, and the classification result is the classification item output after correction. For specific details, please refer to the above, which will not be elaborated here. Please refer to Figure 16 , exemplarily, the text processing module inputs the multi-modal encoding information as the K vector and the V vector into the Transformer Decoder 1501, and the start symbol <s>The Q-vector is input to Transformer Decoder1501 via Output Embedding and Positional Encoding. Figure 17 This is a schematic diagram illustrating the processing flow of a Transformer Decoder. Please refer to... Figure 17 Assuming the number of stacked Transformer Decoder 1501 units Q in this embodiment is 2,

[0261] Optionally, the Output Embedding can be a Word Embedding. The specific implementation can refer to the method in the above embodiment or the implementation method in other existing technical embodiments, which will not be pursued in this application.

[0262] For example, assuming the stacking number Q of Transformer Decoder 1501 in this embodiment is 2 (which can be set according to actual needs, and is not limited in this application), including Transformer Decoder 1501a and Transformer Decoder 1501b. For example, the text processing module inputs multimodal encoding information as K vector and V vector into Transformer Decoder 1501a, and the start symbol... <s>Through Output Embedding and Positional Encoding, it is used as the Q vector to input Transformer Decoder1501a. The input of Transformer Decoder1501a is used as the Q vector input of Transformer Decoder1501b, and the multimodal encoding information is used as the K vector and V vector to input Transformer Decoder1501b. The output of Transformer Decoder1501b is denoted as E dout 1, E dout 1 passes through the fully connected layer to obtain E out 1, where the dimension of E out 1 is (seq_len, N vocab ). Optionally, the text processing module slices the first dimension of E out 1 and takes its last column to obtain a one-dimensional tensor with a length of N vocab . The text processing module passes this one-dimensional tensor through the argmax layer (it can also be the argmax and softmax layers, which can be set according to actual needs, and this application does not make a limitation). Among them, N vocab is optionally the number of texts included in the text sequence table. For example, if the dictionary includes 100 words and corresponding serial numbers, then the value of N vocab is 100. Exemplarily, the value of seq_len is the number of output characters. For example, in the embodiment of this application, the number of output characters is 5, including "fire", "mountain", "explosion", "occurrence" and the end symbol <end>For example, the numerical value output by the argmax layer is used to indicate the ordinal number in the dictionary. The text processing module can determine the corresponding character or word based on the ordinal number. In this example, the text processing module can determine that the corresponding character or word is "fire". That is, the text processing module combines multimodal encoding information and start characters. <s>, the character "fire" can be obtained through the Transformer Decoder 1501.

[0263] Still referring to Figure 16 , the multi-modal encoded information serves as the K vector and the Q vector, and the character "fire" and the start symbol <s>As input to Transformer Decoder1501 as a Q-vector. Optionally, the "fire" character and the start character... <s>The Transformer Decoder1501a is used as the Q-vector input through Output Embedding and Positional Encoding. The Transformer Decoder1501a is based on multimodal encoding information, the "fire" character, and the start character. <s>, output E dout 2. E dout 2 passes through the fully connected layer E out 2. E out 2 obtains the corresponding value through the argmax layer. Based on this value, the text processing module can determine the corresponding character, for example, "mountain". That is to say, the text processing module combines the multi-modal encoded information, the character "fire" and the start symbol <s>The character "mountain" can be obtained through the Transformer Decoder 1501. For details not described, refer to the relevant content for obtaining the character "fire" above, which will not be elaborated here.

[0264] Please continue to refer to Figure 16 , the multi-modal encoded information serves as the K vector and the Q vector, the character "fire", the character "mountain" and the start symbol <s>As Q-vector input to Transformer Decoder1501. Optionally, the "fire" character, the "mountain" character, and the start character... <s>The Transformer Decoder1501a is used as a Q-vector input through Output Embedding and Positional Encoding. The Transformer Decoder1501a is based on multimodal encoding information, the "fire" character, the "mountain" character, and the start character. <s>, output E dout 3. E dout 3 passes through the fully connected layer E out 3. E out 3 obtains the corresponding value through the argmax layer. Based on this value, the text processing module can determine the corresponding character, such as "爆". That is to say, the text processing module combines the multi-modal encoding information, the character "火", the character "山" and the start symbol <s>The character "爆" can be obtained through the Transformer Decoder 1501. Thus, the incorrect character "暴" in the OCR technology recognition result is corrected to "爆". For details not described, refer to the relevant content for obtaining the character "火" above, which will not be elaborated here.

[0265] Please continue to refer to Figure 16 , the multimodal encoded information is used as the K vector and the Q vector, and the characters "火", "山", "爆" and the start symbol <s>As Q-vector input to Transformer Decoder1501. Optionally, the characters "fire", "mountain", and "explosion" are used with the start character. <s>The Transformer Decoder1501a is used as the Q-vector input through Output Embedding and Positional Encoding. The Transformer Decoder1501a is based on multimodal encoding information, the characters for "fire," "mountain," and "explosion," and the start character. <s>with start symbol <s>, output E dout 4. E dout 4 passes through the fully connected layer E out 4. E out 4 obtains the corresponding value through the argmax layer. Based on this value, the text processing module can determine the corresponding character, for example, "发". That is to say, the text processing module combines the multi-modal encoded information, the character "火", the character "山", the character "爆" with the start symbol <s>The character "发" can be obtained through the Transformer Decoder 1501. For details not described, please refer to the relevant content for obtaining the character "火" above, which will not be elaborated here.

[0266] Please continue to refer to Figure 16 , the multi-modal encoded information serves as the K vector and the Q vector, and the characters "火", "山", "爆", "发" and the start symbol <s>As Q-vector input to Transformer Decoder1501. Optionally, the characters "fire", "mountain", "explosion", and "start" are used as start characters. <s>Through Output Embedding and Positional Encoding, it is input into Transformer Decoder 1501a as the Q vector. Transformer Decoder 1501 is based on multimodal encoded information, the characters "fire", "mountain", "explode", "launch", and the start symbol <s>Output E dout 5. E dout 5. Through the fully connected layer E out 5. E out 5. The corresponding numerical value is obtained through the argmax layer. The text processing module determines the output result as the end-of-line character. <end>This means the loop ends.

[0267] For example, after detecting a classification result of 'b' (i.e., the corrected output classification item), the text processing module can obtain the corrected result output by the correction module, namely "volcanic eruption". The text processing module displays the obtained corrected result in the recognition results.

[0268] It should be noted that the models involved in the embodiments of this application include, but are not limited to: image encoding models, text encoding models, multimodal encoding models, classification models, and correction models, which can form a text processing model, or can be understood as a neural network. During the training of the text processing model, the input data mainly consists of text images (including truncated and untruncated samples) and corresponding text recognition content (i.e., text content). For each input model, a text image and its corresponding text content form a pair of training samples. Each pair of training samples can be manually labeled with the three categories mentioned above. That is, the input text images and text content are classified through manual labeling. Specifically, for cases where correction is possible, the text to be corrected is manually modified to obtain the corrected text, which serves as the supervision data output by the text correction decoder. Optionally, the training process of the text processing model is supervised training. The classification decoder (i.e., the classification model) uses classification cross-entropy loss as its time-varying function. The text correction decoder (i.e., the correction model) is trained similarly to the native Transformer autoregressive decoder, using a teacher-forcing method for each time step. Since the two decoders share the encoder (i.e., the backbone of the neural network of the text processing model), the actual training process is joint training.

[0269] In one possible implementation, as described above, truncated text can also include horizontally truncated text and diagonally truncated text. It should be noted that for horizontally truncated text, such as when the first character of each line of text is truncated, in this scenario, the text recognition module, based on OCR recognition, can typically perform prediction and other processing on the text content to obtain the correct text. That is, horizontally truncated text generally does not have the semantic error problem of vertically truncated text as described above. Correspondingly, the scheme in this embodiment can also be aligned and processed when applied to horizontally truncated text. The processed result may have little difference from the recognition result of OCR technology. Similar to horizontally truncated text, for text lines with a small diagonal angle (e.g., less than or equal to 10°), the correct text content can be obtained in OCR technology through prediction and other methods. That is, after processing by the scheme in this embodiment, the output result has little difference from the recognition result of OCR technology. However, for text with a large diagonal angle (e.g., greater than 10°), OCR technology may not be able to recognize all text regions. For example, ... Figure 18a As shown, assuming the angle between text lines is 30°, OCR technology, during text region detection, only identifies the portion of text within the dotted line. When recognizing the text content of the detected region, OCR technology, based on its prediction function, can output text content consistent with the original text. This can also be understood as meaning that for text with a large diagonal angle, the corresponding recognition result may not have semantic errors.

[0270] It should be noted that the technical solution in this application embodiment can effectively solve the problem of semantic errors in the recognition results of partially occluded text. In this application embodiment, "partial occlusion" can optionally mean that the upper part of all characters in the entire line of text is occluded, for example... Figure 4 The scenario shown in (1) involves the first line of text being obscured. In one example, "partial obscuration" could optionally mean that the bottom part of the entire line of text is obscured. For example, Figure 18b This is an illustrative diagram of an application scenario; please refer to it. Figure 18b For example, the image to be recognized includes a text line whose lower portion is truncated. For the OCR recognition result corresponding to this text line, the text processing module can also process it based on the scheme described in the above embodiments. In another example, "partial occlusion" can optionally mean that the upper part (or lower part, or any part) of a portion of the text in a line is occluded. For example, Figure 18c This is an illustrative diagram illustrating another application scenario. Please refer to... Figure 18c In this case, a portion of the text in the image to be recognized is obscured. That is, the original text is "multimodal encoded information (intermediate representation information)," and this "intermediate representation information" is partially obscured. Optionally, the text recognition module performs OCR recognition on this text line, which can obtain multiple text regions. For example... Figure 18d As shown, the text recognition module may identify the text region corresponding to the "multimodal coding information," the text region corresponding to the occluded "(intermediate representation information)," and the text content corresponding to both text regions. Then, the text processing module can execute the processing scheme in this embodiment on the images of the two text regions and their corresponding text content. Optionally, the text recognition module may also obtain a text region by performing OCR recognition on the text line, for example... Figure 18e As shown, the text recognition module may group the occluded text portion and the unoccluded text portion into the same text region. This application embodiment can also process the image and text content of such text regions. That is, the technical solution in this application embodiment can be applied to various scenarios where text is occluded, thereby meeting the text recognition needs in different scenarios. Optionally, this application embodiment can effectively solve the text recognition problem for text lines with an occlusion rate of 20% to 50% (or fluctuating within this range, which is not limited in this application). It should be noted that, as mentioned above, if the occlusion rate of a text line is too high (e.g., 80%), the corresponding text region may not be detected in the OCR stage. If the occlusion rate is low, the OCR recognition result may be correct. The text processing module can directly output or correct the output of the corresponding text content.

[0271] Figure 19 This is a flowchart illustrating another text recognition method provided in an embodiment of this application. Please refer to... Figure 19 This method includes, but is not limited to:

[0272] (1) The text processing module uses a classification model to obtain the classification result of the text image.

[0273] (2) The text processing module determines whether the text content has been truncated based on the classification results.

[0274] For example, the text processing module can preprocess the text image, such as resizing the text image. For details, please refer to the relevant content of the above embodiments, which will not be repeated here.

[0275] For example, Figure 20 For an illustrative illustration of text image processing, please refer to... Figure 20 For example, still using the text image 602a mentioned above, the text processing module inputs the text image 602a (or a processed text image) into the classification model. The classification model can classify the text image 602a and obtain the classification result.

[0276] Optionally, the training data used by the classification model during the training phase includes, but is not limited to, text images corresponding to truncated text and text images corresponding to untruncated text.

[0277] Alternatively, the classification model can be trained under the supervision of the cross-entropy loss function.

[0278] Optionally, the classification model can include, but is not limited to, mainstream classification networks based on Convolutional Neural Networks (CNNs) (such as VGG, ResNet, EfficientNet, etc.), or VIT (Vision Transformer) classification models and their variants based on the Transformer structure. Its primary purpose is to output the probability of a binary classification problem, that is, the score corresponding to either a truncated or untruncated classification term.

[0279] For example, if we denote the classification model as CLS, then the output of the classification model (also known as the classification result) can be expressed as:

[0280] score=CLS(I) (9)

[0281] Here, 'I' is used to indicate a text image, which includes parameters in three dimensions: width, height, and number of channels. For specific concepts, please refer to [reference needed]. Figure 10 The relevant content will not be repeated here.

[0282] Optionally, the output score can be a value greater than 0 or less than 1. The closer the score is to 1, the higher the truncation probability. The text processing module can set a truncation threshold, for example, 0.5, which can be set according to actual needs; this application does not impose a limitation. In one example, if the output score is greater than or equal to the truncation threshold (0.5), the text content corresponding to the text image is determined to be truncated text. In another example, if the output score is less than the truncation threshold (0.5), the text content corresponding to the text image is determined to be untruncated text.

[0283] (3) Output text.

[0284] For example, if the text processing module determines that the text content corresponding to the text image is untruncated text, it can directly output the corresponding text content, that is, display the corresponding text content in the recognition result. For parts not described, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0285] (4) The text processing module uses the semantic model to obtain the semantic judgment result of the text content.

[0286] For example, if the text processing module determines that the text content corresponding to the text image is truncated text, the text processing module will input the text content corresponding to the text image into the semantic model (also known as the semantic judgment module).

[0287] For example, Figure 21 For an illustrative example of the semantic model processing flow, please refer to [link / reference]. Figure 21 The semantic model processing flow includes, but is not limited to, the following steps:

[0288] a. The text processing module segments the text content into words to obtain the segmentation results.

[0289] For example, taking the text content 602b in the above embodiment as an example, the text processing module (specifically the semantic model) segments the text content 602b into words and obtains the corresponding word segmentation sequence. The specific steps for word segmentation and obtaining the text sequence can be found in the relevant content in the above embodiment, and will not be repeated here.

[0290] b. The text processing module processes the word segmentation results through Word embedding and Positional Endcoating to obtain E text .

[0291] For example, the text module (specifically the semantic model) obtains the text encoding information E from the acquired text sequence number through Wordembedding and Positional Endcoating. text For specific details, please refer to the embodiments described above. Figure 12 The relevant descriptions will not be repeated here.

[0292] c. The text processing module will E text F is obtained through the encoding module. text .

[0293] For example, the text processing module will E text Through the encoding module (i.e., the encoder), encoded information with high-dimensional semantic features can be obtained, i.e., F. text The encoding module includes, but is not limited to: CNN encoder, RNN encoder, BiRNN (Bidirectional Recurrent Neural Network) encoder (such as Bidirectional LSTM (Long Short-Term Memory) network), Transformer Encoder, etc., and this application does not limit it. The processing flow of its encoder can be referred to... Figure 14a and Figure 14b The relevant descriptions are not repeated here. In its implementation, E... text Replace with Figure 14a and Figure 14b Multimodal coding information in.

[0294] For example, let the encoder be denoted as Encoder, then F text It can be represented as:

[0295] F text =E ncoder (E text (10)

[0296] d. The text processing module will use F text The output score is obtained through the decoding module. t (That is, the semantic judgment result).

[0297] For example, let the decoding module (i.e., the decoder) be denoted as Decoder, and the score be denoted as _. t It can be represented as:

[0298] score t =Decoder(F text (11)

[0299] Optionally, the decoding module includes, but is not limited to, MLP (fully connected layer) decoders, CNN decoders, RNN decoders, and Transformer decoders. These can be configured according to actual needs, and this application does not impose any limitations. The specific processing flow of the decoding module can be found in [reference needed]. Figure 15 , Figure 16 and Figure 17 The relevant content will not be repeated here. Optionally, since the output result in this example is the score... t This is the result of a binary classification problem. It can be understood that the output indicates semantic coherence or incoherence. Accordingly, the decoder may not include an argmax layer. An argmax layer may also be included in other embodiments, and this application is not limiting.

[0300] For example, in this example, the semantic model's input is primarily a line or a string, and the output is a category (i.e., semantically coherent or semantically incoherent). During training, the semantic model collects corpora and determines semantic coherence for each line through manual annotation. Optionally, the semantic model can also obtain positive and negative training samples through data generation or other methods.

[0301] For example, similar to the classification model, the decoding module outputs a score. t It can be used to indicate semantic coherence. For example, score t Optionally, the threshold value can be greater than 0 or less than 1. The text processing module can set a semantic coherence threshold, such as 0.5. It can be set according to actual needs, and this application does not limit it.

[0302] In one example, if score t If the value is greater than or equal to the semantic coherence threshold (i.e., 0.5), the text processing module can determine the semantic coherence in the corresponding text content. In other words, the OCR technology correctly recognizes the truncated text, and the text processing module can directly output the text content, that is, display the corresponding text content in the text recognition result.

[0303] In another example, if score t If the value is less than the semantic coherence threshold (i.e., 0.5), the text processing module can determine that the corresponding text content is semantically incoherent. In other words, if there is a semantic error in the recognition result of OCR technology for truncated text, the text processing module continues to execute step (5).

[0304] It should be noted that the semantic coherence model used in this embodiment to detect text semantic coherence is merely an illustrative example. In other embodiments, the text processing module can also detect semantic coherence in other ways. For example, it can be based on a syntax error detection model, which can output a candidate set of positions of syntax errors based on the input text content, and set a threshold based on the ratio of the candidate set to the total number of tokens (minimum semantic units). As another example, the text processing module can use a forward language model to obtain the probability of each token and make a judgment based on the average probability and a preset threshold. Specific details can be found in the relevant content of existing technical embodiments, and will not be repeated here.

[0305] (5) The text processing module determines whether the text content can be corrected.

[0306] In this embodiment, the text processing module can further determine whether the text content can be corrected based on the results output by the semantic model. For example, the text processing module can set a correction threshold, such as 0.2, which can be set according to actual needs and is not limited in this application.

[0307] In one example, if score t If the value is greater than or equal to the correction threshold (i.e., 0.2), the text processing module can determine that the corresponding text content is correctable, and can then output the corrected text content. For example, the text processing module can use the text content as input to the correction module for correction. The processing flow of the correction module can be found in [reference needed]. Figure 15 , Figure 16 and Figure 17 The relevant content will not be repeated here.

[0308] In another example, if score t If the value is less than the correction threshold (i.e., 0.2), the text processing module can determine that the corresponding text content cannot be corrected. Therefore, the text processing module filters the text content and does not display the corresponding text content in the text recognition results.

[0309] It should be noted that the method of determining whether text content can be corrected in step (5), i.e., the detection method based on the output of semantic coherence, is only an illustrative example. In other embodiments, the text processing module can also use other detection methods to detect whether the text content can be corrected. For example, as mentioned above, the text processing module can process the semantic coherence judgment based on the syntax error detection model. The text processing module can further determine the number of syntax errors or the proportion of syntax error characters based on the output of the syntax error detection model. The text processing module can determine whether the text content can be corrected based on this proportion. As another example, as mentioned above, the semantic coherence judgment can be based on the average probability calculated by the forward language model. The text processing module can determine whether the text content can be corrected based on this average probability (e.g., by setting a corresponding correction threshold).

[0310] It should be further noted that, in addition to correcting the text content using the correction method described in the embodiments of this application (which can also be understood as a neural machine translation method), the text processing module can also employ other correction methods. For example, it can correct the text content based on the output of the syntax error detection model, using confusion set recall and candidate ranking. As another example, the text processing module can obtain a confusion set of error locations based on the output of the syntax error detection model by calling a statistical language model, a neural language model, or BERT's bidirectional language model, and then recall the corrected text through candidate ranking and error distance mechanisms. Specific implementation details can be found in the relevant content of existing technical embodiments, and will not be elaborated here.

[0311] It should be further explained that, Figure 19 The various models involved in each step can form a neural network. The training method of this neural network can be referred to the relevant description of neural network training in the above embodiment, which will not be repeated here.

[0312] In one example, Figure 22 The schematic block diagram illustrating an embodiment of the present application shows an apparatus 2200. The apparatus 2200 may include a processor 2201 and a transceiver / transceiver pin 2202, and optionally, a memory 2203.

[0313] The various components of device 2200 are coupled together via bus 2204, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus 2204 in the figure.

[0314] Optionally, the memory 2203 can be used for the instructions in the foregoing method embodiments. The processor 2201 can be used to execute the instructions in the memory 2203, control the receive pin to receive signals, and control the transmit pin to transmit signals.

[0315] Device 2200 may be an electronic device or a chip of an electronic device as described in the above method embodiments.

[0316] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0317] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the methods described in the above embodiments.

[0318] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the method described in the above embodiment.

[0319] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the methods in the above-described method embodiments.

[0320] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0321] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0322] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0323] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0324] In the embodiments of this application, the words "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0325] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0326] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.< / end> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / end> < / s> < / s>

Claims

1. A text recognition method, characterized in that, include: The electronic device performs text region detection on the object to be identified, and obtains an image of the first text region; The first text area includes text content; The electronic device performs text content recognition on the first text region to obtain the first text content; The electronic device classifies the image of the first text region and the content of the first text to obtain a classification result. Based on the classification results, the electronic device displays the text recognition results for the first text region; Based on the classification results, the electronic device displays the text recognition results for the first text region, including: If the classification result is the first category, the text recognition result filters out the first text content; If the classification result is the second category, the text recognition result includes the text content after the first text content has been corrected; if the classification result is the third category, the text recognition result includes the first text content.

2. The method according to claim 1, characterized in that, The electronic device classifies the image of the first text region and the first text content to obtain a classification result, including: The electronic device obtains intermediate representation information based on the image of the first text region and the first text content; The electronic device classifies the intermediate representation information to obtain the classification result.

3. The method according to claim 2, characterized in that, The electronic device classifies the intermediate representation information to obtain the classification result, including: The electronic device classifies the intermediate representation information using a classification model to obtain the classification result.

4. The method according to claim 3, characterized in that, Before the electronic device displays the text recognition result of the first text region based on the classification result, it further includes: The electronic device corrects the intermediate representation information to obtain the corrected text content of the first text content.

5. The method according to claim 4, characterized in that, The electronic device corrects the intermediate representation information to obtain the corrected target text content, including: The electronic device corrects the intermediate representation information using a correction model to obtain the corrected text content of the first text content.

6. The method according to claim 5, characterized in that, The electronic device obtains intermediate representation information based on the image of the first text region and the first text content, including: The electronic device performs image encoding on the image of the first text region to obtain first image encoding information; The electronic device performs text encoding on the first text content to obtain first text encoding information; The electronic device performs multimodal encoding on the first image encoding information and the first text encoding information using a multimodal encoding model to obtain the intermediate representation information.

7. The method according to claim 6, characterized in that, The multimodal coding model, the classification model, and the correction model constitute a neural network. The training data of the neural network includes a second text region and the second text content corresponding to the second text region, as well as a third text region and the third text content corresponding to the third text region. The second text region includes partially missing text content, and the text content in the third text region is complete text content.

8. The method according to claim 1, characterized in that, The text recognition result of the first text region is displayed in the text recognition area, which also includes the text content corresponding to the third text region of the object to be recognized.

9. The method according to claim 1, characterized in that, If the first text region includes partially missing text content, the text recognition result is either the first category or the second category.

10. The method according to claim 9, characterized in that, The semantics expressed by the first text content are different from the semantics expressed by the text content in the first text region.

11. The method according to any one of claims 1 to 10, characterized in that, The object to be identified is an image, webpage, or document.

12. A text recognition method, characterized in that, include: The electronic device performs text region detection on the object to be identified, and obtains an image of the first text region; The first text area includes text content; The electronic device performs text content recognition on the first text region to obtain the first text content; The electronic device displays the text recognition result of the first text region based on the image of the first text region and the first text content; The electronic device displays the text recognition result of the first text region based on the image of the first text region and the first text content, including: If the image representation of the first text region indicates that the first text region includes partially missing text content and the first text content is semantically coherent, or if the image representation of the first text region indicates that the first text region does not include partially missing text content, the text recognition result includes the first text content; if the image representation of the first text region indicates that the first text region includes partially missing text content and the first text content includes semantically incorrect text content, the text recognition result filters out the first text content or the text recognition result includes the corrected text content.

13. The method according to claim 12, characterized in that, The electronic device displays the text recognition result of the first text region based on the image of the first text region and the first text content, including: If the image representation of the first text region includes partially missing text content, and the first text content includes semantically incoherent text content, the electronic device detects whether the first text content can be corrected. If the first text content cannot be corrected, the text recognition result filters out the first text content; If the first text content can be corrected, the text recognition result includes the corrected text content.

14. The method according to claim 13, characterized in that, If the first text content can be corrected, the method further includes: The electronic device corrects the first text content using a correction model to obtain the corrected text content.

15. The method according to claim 14, characterized in that, The electronic device displays the text recognition result of the first text region based on the image of the first text region and the first text content, including: The electronic device classifies the image of the first text region using a classification model to obtain a classification result; the classification result is used to indicate whether the first text region includes partially missing text content.

16. The method according to claim 15, characterized in that, If the image of the first text region represents that the first text region includes partially missing text content, the electronic device displays the text recognition result of the first text region based on the image of the first text region and the first text content, including: The electronic device performs semantic analysis on the first text content using a semantic model to obtain semantic analysis results; the semantic analysis results are used to indicate whether the first text content includes semantically incorrect text content.

17. The method according to claim 16, characterized in that, The semantic analysis result is also used to indicate whether the first text content can be corrected. The electronic device displays the text recognition result of the first text region based on the image of the first text region and the first text content, including: Based on the semantic analysis results, the electronic device determines whether the first text content can be modified.

18. The method according to claim 17, characterized in that, The correction model, the classification model, and the semantic model constitute a neural network. The training data of the neural network includes a second text region and the second text content corresponding to the second text region, as well as a third text region and the third text content corresponding to the third text region. The second text region includes partially missing text content, and the text content in the third text region is complete text content.

19. The method according to claim 12, characterized in that, The text recognition result of the first text region is displayed in the text recognition area, which also includes the text content corresponding to the third text region of the object to be recognized.

20. The method according to claim 19, characterized in that, The semantically incorrect text content expresses a semantic meaning that is different from the semantic meaning of the corresponding text content in the first text region.

21. The method according to any one of claims 12 to 20, characterized in that, The object to be identified is an image, webpage, or document.

22. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 11.

23. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 12 to 21.

24. A computer-readable storage medium, characterized in that, Includes a computer program that, when the computer program is running on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 11.

25. A computer-readable storage medium, characterized in that, Includes a computer program that, when the computer program is run on an electronic device, causes the electronic device to perform the method as described in any one of claims 12 to 21.

26. A computer program product, characterized in that, The method includes a computer program that, when executed by an electronic device, causes the electronic device to perform the method according to any one of claims 1 to 11.

27. A computer program product, characterized in that, Includes a computer program, which, when executed by an electronic device, causes the electronic device to perform the method of any one of claims 12 to 21.

Images