Text recognition method, device, product and medium

Abstract

The application applies to the field of artificial intelligence, and provides a character recognition method, device, product and medium, wherein the method comprises: acquiring a to-be-recognized image containing characters; determining a first text line detection box and a plurality of text block detection boxes in the to-be-recognized image; determining that the characters in the first text line detection box belong to the plurality of text block detection boxes in correspondence to the overlap of the first text line detection box and the plurality of text block detection boxes; and dividing the first text line detection box into a plurality of second text line detection boxes, wherein each second text line detection box only overlaps with one text block detection box. In this way, the text line detection box can be corrected through the text block detection box, and the characters in each text line detection box only belong to one column. The problem that the order of the recognized characters is inconsistent with the order of the characters in the to-be-recognized image due to the recognition of the characters in different columns as the same line of characters is avoided, and the accuracy of the character recognition result is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, device, product and medium for character recognition. Background Technology

[0002] Text recognition technology can identify and extract text from images containing text. Currently, text recognition technology typically extracts text line by line, meaning that multiple text line detection boxes are determined based on the size of each line of text, with each detection box corresponding to one line of text. The electronic device then crops an image of each line of text based on each detection box and feeds the image of each line of text into a text recognition model to identify the text content.

[0003] However, if the text in an image is laid out in multiple columns, such as a two-column or three-column layout, the text line detection box may mistakenly detect text in the same line from adjacent columns as text from the same column. This results in the recognized text order being inconsistent with the original image layout, making the recognition results unreadable and impacting the user experience. Summary of the Invention

[0004] In view of this, this application provides a character recognition method, device, product, and medium.

[0005] In a first aspect, a text recognition method is provided, comprising: acquiring an image to be recognized containing text; determining a first text line detection box and multiple text block detection boxes in the image to be recognized; determining that the text in the first text line detection box belongs to multiple text block detection boxes corresponding to the overlap between the first text line detection box and the multiple text block detection boxes; dividing the first text line detection box into multiple second text line detection boxes, wherein each second text line detection box overlaps with only one text block detection box.

[0006] In the above scheme, the first text line detection box corresponds to a line of text in the image to be recognized, and each text block detection box corresponds to a text block. The first text line detection box is determined based on the size of the text line, and the text block detection box is determined based on the distribution of paragraphs in the text. Each text block detection box corresponds to only part or all of the text in a column.

[0007] In this way, text block detection boxes can correct errors in text line detection boxes, ensuring that the text in each text line detection box belongs to only one column. This avoids the problem of recognizing text from different columns as the same line, leading to inconsistencies between the recognized text order and the text order in the image, thus improving the accuracy of text recognition results.

[0008] In conjunction with the first aspect, in some implementations, the method further includes: determining the size of the overlapping area between the first text line detection box and each text block detection box; and determining that the first text line detection box overlaps with the multiple text block detection boxes when the size of the overlapping area between the first text line detection box and the multiple text block detection boxes is greater than a threshold.

[0009] In the above scheme, the area of ​​the overlapping region between the text line detection box and the text block detection box is calculated to determine whether the text line detection box has detected text from multiple columns as a single line. When the size of the overlapping region between the text line detection box and the text block detection box is greater than a threshold, it can be considered that the text line detection box has recognized and detected the text in that text block detection box. When the text line detection box overlaps with multiple text block detection boxes, it can be considered that the text line detection box has detected text from multiple columns.

[0010] This approach identifies cases where text from multiple columns is detected as the same line of text. It allows for the splitting of such text line detection boxes, ensuring that the text within each detection box belongs to only one column. This avoids the problem of misidentifying text from different columns as the same line, leading to discrepancies between the recognized text order and the actual text order in the image, thus improving the accuracy of text recognition results.

[0011] In conjunction with the first aspect, in some implementations, the method further includes: determining that the text in the first text line detection box belongs to only one text block detection box when the size of the overlapping area between the first text line detection box and only one text block detection box is greater than a threshold.

[0012] In the above scheme, when a text line detection box overlaps with only one text block detection box, it can be determined that the text line detection box only recognizes the text of one column. In this way, the text in each text line detection box belongs to only one column. This avoids the problem of recognizing text from different columns as the same line, leading to inconsistencies between the recognized text order and the text order in the image to be recognized, thus improving the accuracy of the text recognition results.

[0013] In conjunction with the first aspect, in some implementations, the method further includes: determining multiple intersection points between the first text line detection box and multiple overlapping text block detection boxes; and dividing the first text line detection box into multiple second text line detection boxes based on the multiple intersection points.

[0014] In conjunction with the first aspect, in some implementations, the method further includes: using the line connecting two intersection points of the same text block detection box among multiple intersection points as the dividing line to obtain multiple dividing lines; and dividing the first text line detection box into multiple second text line detection boxes based on the multiple dividing lines.

[0015] In the above scheme, the dividing line of the text line detection box is determined based on the intersection of the text line detection box and the text block detection box. This ensures that the text in each text line detection box belongs to only one column. This avoids the problem of misidentifying text from different columns as the same line, leading to inconsistencies between the recognized text order and the text order in the image, thus improving the accuracy of text recognition results.

[0016] In conjunction with the first aspect, in some implementations, the method also includes: performing text recognition based on multiple second text line detection boxes to obtain multiple line recognition results.

[0017] In the above scheme, after determining that the text in each text line detection box belongs to only one column, the text in the text line detection box is recognized, resulting in multiple line recognition results. This avoids the problem of the recognized text order being inconsistent with the text order in the image to be recognized due to recognizing text from different columns as the same line, thus improving the accuracy of the text recognition results.

[0018] In conjunction with the first aspect, in some implementations, the method further includes: determining the type of the image to be identified; and determining the size of the overlapping area between the first text line detection box and each text block detection box only when the image to be identified belongs to a document image, wherein the document image includes text in multiple columns.

[0019] In the above scheme, the type of the image to be recognized will also be determined. Document images are more likely to have columns. Only when the image to be recognized is a document image will the text line detection box determine whether multiple columns of text are recognized as the same. In this way, the above steps are not required for images that are not document images, which can improve the efficiency of text recognition.

[0020] In a second aspect, this application provides an electronic device including a processor and a memory, wherein the memory is used to store instructions and the processor is used to execute the instructions, and when the processor executes the instructions, it performs the method described in the first aspect.

[0021] Thirdly, this application provides a computer-readable storage medium storing instructions that, when executed on an electronic device, perform the method described in the first aspect.

[0022] Fourthly, this application provides a computer program product including computer instructions, which, when executed by a computing device, cause the computing device to perform the method described in the first aspect.

[0023] In summary, the text recognition method, device, product, and medium provided in this application can correct text line detection boxes based on text block detection boxes, ensuring that the text in each text line detection box belongs to only one column. This avoids the problem of incorrect text order caused by recognizing text from different columns as the same line, resulting in inconsistencies with the text order in the original image. Therefore, it improves the accuracy and readability of the text recognition results. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0025] Figure 1 This is a schematic diagram of the text line detection box and text block detection box provided in the embodiments of this application;

[0026] Figure 2 This is a schematic diagram of the column-based text display provided in an embodiment of this application;

[0027] Figure 3 This is a schematic diagram of the text line detection box and text block detection box in the column scenario provided in the embodiments of this application;

[0028] Figure 4 This is a schematic diagram illustrating the segmentation of text line detection boxes based on text block detection boxes in a column-based scenario provided in this application embodiment;

[0029] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0030] Figure 6 This is a flowchart illustrating the text recognition method provided in an embodiment of this application;

[0031] Figure 7 This is a schematic flowchart of the method for adjusting the text line detection box provided in the embodiments of this application;

[0032] Figure 8 This is a schematic diagram of the overlapping area of ​​the text line detection box and the text block detection box provided in the embodiments of this application;

[0033] Figures 9A to 9B This is a schematic diagram of the split text line detection box provided in an embodiment of this application;

[0034] Figure 10 This is a schematic diagram of the character recognition device provided in this application. Detailed Implementation

[0035] The illustrative embodiments of this application include, but are not limited to, character recognition methods, devices, products, and media.

[0036] To facilitate understanding, the application scenarios involved in this application are described below with reference to the accompanying drawings.

[0037] In an increasing number of scenarios, text recognition in images is required. For example, in photo translation applications, electronic devices need to first recognize the text in the image before translating it. Similarly, when digitizing and backing up paper documents, an electronic device can first photograph the document, recognize the text, and then back it up.

[0038] Currently, electronic devices typically employ optical character recognition (OCR) technology to identify text in images. OCR technology includes text detection and text recognition processes. A text detection model detects each line of text in the image, and then a text recognition model identifies the characters within each line. To improve the readability of the recognition results, paragraph analysis is performed after text recognition, ensuring that the output recognition result can reconstruct the text layout from the original image.

[0039] Specifically, the electronic device first uses a text detection model to determine the detection boxes for each line of text in the image, with each detection box corresponding to a line of text. Then, the electronic device uses the coordinate information of the text detection boxes to crop the image of each line of text, and finally, the cropped image is fed into the text recognition model to recognize the content of each line of text.

[0040] Electronic devices also use layout segmentation models to detect features such as the spacing between each line of text and whether there are blank spaces before and after each line of text. They also use features such as smaller line spacing within a paragraph than between paragraphs, blank spaces at the beginning of a paragraph, and blank spaces at the end of a paragraph to analyze the distribution of each paragraph in the image and determine multiple non-overlapping text block detection boxes. Each text block detection box corresponds to a part of the text in the image.

[0041] It should be understood that, due to text layout, text within the same paragraph may not be displayed continuously in the same area; that is, the text within the same paragraph may be divided into multiple text blocks. Therefore, the text within the same paragraph may correspond to multiple text block detection boxes.

[0042] Finally, the electronic device determines the text block detection box corresponding to each text line detection box, which allows the recognition results of one or more lines belonging to the same text block to be merged together in sequence to obtain the final output recognition result.

[0043] For example, such as Figure 1 As shown, Figure 1 This example illustrates a diagram of text line detection boxes and text block detection boxes during OCR text recognition. Figure 1As shown in (A), the electronic device obtained multiple text line detection boxes, including box A1 to box A6, through a text detection model. Each text line detection box corresponds to a line of text. Figure 1 As shown in (B), the electronic device obtains multiple text blocks, including block A1 and block A2, through the page segmentation model. Each text block detection box corresponds to a paragraph. Therefore, it can be determined that boxes A1 to A3 belong to block A1, and boxes A4 to A6 belong to block A2.

[0044] However, in some document images, text is often displayed in a multi-column layout. For example, a document might look like this: Figure 2 As shown in (A), it is displayed in a left-right two-column layout, or as shown in... Figure 2 As shown in (B), the text is displayed in three columns: left, middle, and right.

[0045] Text detection models might mistakenly classify closely spaced lines of text in adjacent columns as the same line. Then, after the layout segmentation model detects text blocks, it might assign a single line of text, originally belonging to multiple blocks, to the block with the largest overlap with the detection box of that line. In the final output, this text block might contain content from other blocks, leading to errors in the recognition result. Furthermore, this can affect subsequent processing, causing translation errors in translation scenarios and semantic analysis errors in natural language processing.

[0046] For example, such as Figure 3 As shown, taking a document in an image with a three-column layout (left, center, right) as an example, the text detection model might mistake text belonging to two columns for the same line when detecting text lines in the image. For example... Figure 3 Both boxB1 and boxB2 detect the text of the corresponding lines on the left and in the middle as a single line of text. In the page segmentation model, four text blocks, including blocks B1 to B4, are detected. When determining the text block corresponding to each text line detection box, boxB1 and boxB2 are assigned to the text line detection box belonging to either blockB1 or blockB2. If they correspond to blockB1, when merging the line recognition results of the various text line detection boxes for blockB1, the recognition result for blockB1 becomes “…ABCDEFGHI ABCDEFGHI JKLMNOPQR JKLMNOPQR…”.

[0047] In other words, current OCR text detection models detect text line by line, and then use the text block recognition results from the layout segmentation model to reconstruct the text layout. However, when text is displayed in multiple columns, the text detection model may mistakenly detect text from the same line in adjacent columns as text from the current column, leading to incorrect text order and thus incorrect recognition results, resulting in a poor user experience.

[0048] To address the issue of misclassifying text from different columns as the same line in multi-column layouts, leading to incorrect recognition results, this application provides a text recognition method applied to an electronic device 100. The electronic device 100 determines multiple non-overlapping text line detection boxes based on the image to be recognized, where each text line detection box corresponds to text from a different line. Furthermore, based on the text layout in the image to be recognized, multiple non-overlapping text block detection boxes are determined, where each text block detection box corresponds to a text block, that is, each text block detection box corresponds to part or all of a paragraph, and each text block detection box also corresponds to part or all of a column.

[0049] Then, based on the degree of overlap between each text line detection box and the text block detection box, it is determined whether a text line detection box overlaps with multiple text block detection boxes, thereby determining whether there are cases where text from different columns is detected as the same line. If a text line detection box overlaps with multiple text block detection boxes, the electronic device 100 will divide the text line detection box into multiple text line detection boxes, ensuring that each text line detection box overlaps with only one text block detection box.

[0050] In this way, the electronic device 100 can correct errors in the text line detection boxes by using text block detection boxes, ensuring that the text in each text line detection box belongs to only one column. This avoids the problem of the recognized text order being inconsistent with the text order in the original image due to detecting text from different columns as the same line of text, thereby improving the accuracy and readability of the text recognition results.

[0051] Optionally, the text in the image to be recognized can be, for example, Figure 2 The column layout shown can specifically be a two-column or three-column arrangement. It should be understood that the layout of the text in the image to be recognized can include even more columns. Alternatively, the size of each column can be different; for example, the document can be displayed with columns offset to the right or left, meaning the right column is larger than the left column, or vice versa. This application does not specifically limit the specific column arrangement of the text in the image to be recognized.

[0052] Optionally, the electronic device 100 determines whether a text line detection box overlaps with multiple text block detection boxes by calculating the size of the overlap area between each text line detection box and a text block detection box using the coordinates of the text line detection box and the coordinates of the text block detection boxes. If the size of the overlap area between a text line detection box and two or more text block detection boxes is greater than a threshold, then the text line detection box is considered to overlap with multiple text block detection boxes.

[0053] Optionally, the electronic device 100 may also determine the intersection point based on the coordinates of the text line detection box and the text block detection box, and segment the text line detection box based on the intersection point, thereby dividing the text line detection box into multiple text line detection boxes, so that each text line detection box overlaps with only one text block detection box.

[0054] For example, refer to the aforementioned Figure 3 As shown, when both boxB1 and boxB2 detect the text in the corresponding lines on the left and in the middle as a single line of text, according to the text recognition method provided in this application, the electronic device 100 will obtain the overlap between each text line detection box and the text block detection box. When the electronic device 100 determines that boxB1 overlaps with both blockB1 and blockB2, and boxB2 also overlaps with both blockB1 and blockB2, that is, boxB1 and boxB2 detect the text in different columns as the same line of text, the electronic device 100 will segment boxB1 and boxB2 according to the coordinate information of blockB1 and blockB2, as follows: Figure 4 As shown, boxB1 is divided into boxB11 and boxB12, and boxB2 is divided into boxB21 and boxB22. Thus, boxB11 and boxB21 only overlap with blockB1, and boxB12 and boxB22 only overlap with blockB2.

[0055] The aforementioned electronic device 100 can be any device with a display screen and / or camera, such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), or specialized cameras (such as SLR cameras, point-and-shoot cameras), etc.

[0056] Electronic device 100 can also be a physical server, such as an x86 server, an ARM server, etc., or a virtual machine (VM) implemented based on a general-purpose physical server combined with network functions virtualization (NFV) technology. A virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. This application does not impose any restrictions on the specific type of electronic device 100.

[0057] The electronic device 100 involved in the embodiments of this application will be introduced first below.

[0058] See Figure 5 , Figure 5 A schematic diagram of the structure of an exemplary electronic device 100 provided in an embodiment of this application is shown.

[0059] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, sensor module 180, button 190, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc.

[0060] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0061] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0062] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, text recognition, facial recognition, and speech recognition.

[0063] In some embodiments of this application, the processor 110 can be used to trigger text recognition of an image based on user operation or detection of preset recognition conditions.

[0064] The charging management module 140 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger.

[0065] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0066] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0067] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network.

[0068] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0069] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including UWB, wireless local area networks (WLAN) (such as wireless fidelity (WiFi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc.

[0070] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with networks and other devices through wireless communication technology.

[0071] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0072] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0073] In some embodiments of this application, the display screen 194 displays the interface content currently output by the system. For example, the display screen 194 can be used to display images from a gallery, display the interface of an application, etc. The display screen 194 can also display text content obtained by the processor 110 after recognizing the image.

[0074] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0075] The ISP is used to process the data fed back by the camera 193.

[0076] Camera 193 is used to capture still images or videos. An object passes through the lens to generate an optical image that is projected onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0077] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0078] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0079] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0080] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0081] Electronic device 100 can implement audio functions such as music playback and recording through audio module 170, speaker, receiver, microphone, headphone jack, and application processor.

[0082] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0083] The sensor module 180 may include pressure sensors, gyroscope sensors, barometric pressure sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0084] A touch sensor, also known as a "touch panel," can be located on the display screen 194. The touch sensor and display screen 194 together form a touchscreen, also called a "touchscreen." The touch sensor detects touch operations applied to or near it, such as actions by the user's hand, elbow, stylus, etc., touching the display screen 194. The touch sensor transmits the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194.

[0085] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0086] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with or separate from the electronic device 100.

[0087] It is understood that the structure illustrated in this application does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0088] The following describes the steps and flow of a character recognition method provided in this application, such as... Figure 6 As shown, the method may include:

[0089] S610: Acquire the image to be recognized.

[0090] The image to be recognized acquired by the electronic device 100 includes images captured by screenshots, images captured by a camera, or images acquired from external sources, such as images downloaded or received by the user via a network using the electronic device 100. The image to be recognized includes text content.

[0091] Optionally, the image to be recognized may also be a portion of the interface displayed by the electronic device 100. That is, the user selects the image to be recognized on the interface of the electronic device 100 through a content selection operation, wherein the content selection operation includes the user determining the image to be recognized by using a body part (e.g., a finger) or an input device (e.g., a stylus, keyboard, mouse, etc.).

[0092] For example, similar to the selection operation of selecting an interface area in a screenshot operation, the electronic device 100 obtains the start position and end position of the content selection operation performed by the user in the display interface, and then uses the start position and end position as the vertices of the polygon to determine the polygon area. The interface corresponding to the polygon area is the image to be recognized.

[0093] Optionally, after the electronic device 100 acquires the image to be recognized, it can also identify the category of the image. The categories of the image to be recognized include document images, screenshot images, portrait images, etc., among which document images have a higher probability of containing text than other image types. In some embodiments, the electronic device only executes subsequent steps S620 and S630 when the image belongs to the category of document images.

[0094] Optionally, the image to be recognized may contain irregularities such as curved surfaces, slope distributions, wrinkles, deformations, or incompleteness. After acquiring the image to be recognized, the electronic device 100 can also perform preprocessing on the image. For example, preprocessing operations may include one or more of the following: standardization, normalization, binarization, contrast-limited adaptive histogram equalization (CLAHE), etc.

[0095] Standardization processing refers to correcting the image to be recognized to conform to a preset standard state when there is a deviation between the image and the standard state. For example, if the image to be recognized is detected to be tilted or upside down, its orientation must be corrected first. Normalization processing refers to representing each pixel value in the image to be recognized using a value in the range [0,1] based on the maximum and minimum pixel values. Binarization processing refers to representing the value of each pixel in the image to be recognized using either 0 or 1. CLAHE can change the image contrast by calculating the local histogram of the image and then redistributing the brightness, thereby obtaining more image details in subsequent text recognition feature extraction.

[0096] S620: Determine multiple text line detection boxes based on the image to be recognized.

[0097] The electronic device 100 will detect the scale information of each line of text in the image to be recognized using a text detection model. The scale information can be in pixels, and its value can be the height and width of the rectangular area occupied by the text, or the area of ​​the rectangular area. Then, the electronic device 100 determines multiple text line detection boxes based on the scale information of each line of text, with each text line detection box corresponding to a line of text in the image to be recognized.

[0098] For a description of the text line detection box, please refer to the aforementioned [link / reference]. Figure 1 , Figure 3 The details and related descriptions will not be repeated here.

[0099] S630: Generates multiple text block detection boxes based on the layout of the image to be recognized.

[0100] Electronic device 100 acquires the text layout information of the image to be recognized, thereby obtaining the distribution of each paragraph in the image and determining multiple non-overlapping text block detection boxes, each text block detection box corresponding to a segment of text in the image to be recognized.

[0101] The text formatting information includes the spacing between each line of text and whether there are spaces before or after each line. The electronic device 100 analyzes the region corresponding to each paragraph of the electronic device based on features such as smaller line spacing within a paragraph compared to line spacing between paragraphs, spaces at the beginning of paragraphs, and spaces at the end of paragraphs. For a description of the text block detection box, please refer to the aforementioned... Figure 1 , Figure 3 The details and related descriptions will not be repeated here.

[0102] In some embodiments, step S630 may be performed before step S620, or steps S620 and S630 may be performed simultaneously. This application does not impose a specific limitation on the execution order of steps S620 and S630.

[0103] S640: Adjust multiple text line detection boxes based on multiple text block detection boxes.

[0104] The electronic device 100 will sequentially calculate the overlap between each text line detection box and each text block detection box, and then determine whether any text line detection box overlaps with multiple text block detection boxes based on the overlap degree. If a text line detection box overlaps with multiple text block detection boxes, it can be determined that the text line detection box has detected text from different columns as the same line of text. The electronic device 100 will then divide the text line detection box into multiple text line detection boxes, ensuring that each of the divided text line detection boxes overlaps with only one text block detection box.

[0105] For a description of adjusting multiple text line detection boxes based on multiple text block detection boxes, please refer to [link / reference]. Figure 1 , Figure 3 And related descriptions, in detail, will be combined later. Figure 8 A detailed description will not be provided here.

[0106] S650: Recognizes the text in each text line detection box to obtain multiple line recognition results.

[0107] When the electronic device 100 determines that each text line detection box corresponds to only one line of text in one column, the electronic device 100 will recognize the text in each text line detection box and obtain the text in each text line detection box, that is, generate the line recognition result corresponding to each text line detection box.

[0108] For example, the electronic device 100 may use a neural network (e.g., a convolutional neural network) to extract the shape features of the characters in each text line detection box. For instance, it may determine the shape of each character in the text line detection box by detecting the dark and light conditions within each text line detection box. Then, a classifier is used to classify the shape of each character, and the classification result with the highest confidence is used as the recognition result for each character in the text line detection box, thereby recognizing the characters in each text line detection box. It should be understood that the embodiments of this application do not limit how the characters corresponding to the given characters are identified.

[0109] S660: Combine multiple line recognition results according to their corresponding text block detection boxes to obtain the recognition result.

[0110] After obtaining the line recognition result corresponding to each text line detection box, the electronic device 100 will combine the line recognition results in sequence according to the position information of the text line detection box and / or the position information of the text block detection box to obtain the final recognition result.

[0111] Optionally, after sequentially combining the line recognition results of the text line detection boxes contained in each text block detection box, and obtaining the text block recognition result corresponding to each text block detection box, the electronic device 100 will also sort the various text blocks. Then, the recognition results of each text block are combined according to the order of the text blocks to obtain the recognition result of all the text in the image to be recognized.

[0112] The text blocks can be ordered in the default top-to-bottom or left-to-right manner, or they can be ordered based on the recognition results of each text block, performing semantic analysis on each text block, and then ordering the text blocks according to the results of the semantic analysis. This application does not specify a particular ordering method for the text blocks.

[0113] Optionally, after obtaining the recognition result of the image to be recognized, the electronic device 100 may also display the recognition result on the display screen 194. Alternatively, the electronic device 100 may also provide related services based on the recognition result, such as word segmentation, semantic analysis, translation, etc. Or, the electronic device 100 may send the recognition result to other devices for further processing.

[0114] For example, after obtaining the recognition result, the electronic device 100 can display the recognition result in a split-screen window or as a mask on the image to be recognized, so that the user can copy, search, translate, or share one or more characters, words, or punctuation marks. The aforementioned mask refers to a separate, complete layer that is overlaid on the current display interface, for example, in the form of a pop-up window. In implementation, the mask can be displayed on the image to be recognized in a transparent, semi-transparent, or opaque manner.

[0115] In summary, the electronic device 100 can correct text line detection boxes by using text block detection boxes, ensuring that the text in each text line detection box belongs to only one column. This avoids the problem of incorrect text order caused by detecting text from different columns as the same line, thereby improving the accuracy of text recognition results.

[0116] The following is combined with Figure 7 The process of adjusting the text line detection box in step S640 above will be described in detail, specifically using the first text line detection box among multiple text line detection boxes as an example. It should be understood that each text line detection box will execute the following steps S710 to S760.

[0117] S710: Get the coordinates of the first text line detection box.

[0118] The electronic device 100 can establish a coordinate system based on the image to be recognized, and determine the position of the first text line detection box based on the coordinates of each vertex of the first text line detection box. The first text line detection box is one of the multiple text line detection boxes in step S620.

[0119] S720: Get the coordinates of each text block detection box.

[0120] The electronic device 100 can obtain the coordinates of each vertex of each text block detection box based on the multiple text block detection boxes determined in step S630. It should be understood that the coordinates of the first text line detection box and each text block detection box in steps S710 and S720 need to be obtained in the same coordinate system.

[0121] In some embodiments, step S720 may be performed before step S710, or steps S710 and S720 may be performed simultaneously. This application does not impose a specific limitation on the execution order of steps S710 and S720.

[0122] S730: Determine the overlap between the first text line detection box and each text block detection box.

[0123] The electronic device 100 calculates the size of the overlapping area between the first text line detection box and each text block detection box based on the coordinates of the first text line detection box and each text block detection box. The size of the overlapping area is the degree of overlap between the first text line detection box and each text block detection box.

[0124] In some embodiments, after calculating the size of the overlapping area between the first text line detection box and each text block detection box, the electronic device 100 will also calculate the ratio of the size of each overlapping area to the area of ​​the first text line detection box, or calculate the ratio of the size of each overlapping area to the area of ​​all overlapping areas, and then use the above ratio as the degree of overlap.

[0125] For example, with Figure 3 Taking boxB2 as an example, which is the detection box for the first text line, as an example, Figure 8 As shown, the overlap between boxB2 and blocks B1 to B4 will be calculated sequentially. The overlap between boxB2 and blockB1 is equal to the area of ​​region 1, and the overlap between boxB2 and blockB2 is equal to the area of ​​region 2. There is no overlap between boxB2 and blocks B3 and B4. Therefore, it can be calculated that region 1 occupies 0.5 of the total area of ​​regions 1 and 2, and region 2 occupies 0.5 of the total area of ​​regions 1 and 2. The overlap between boxB2 and blocks B1 to B4 can be found in Table 1 below.

[0126] blockB1 blockB2 blockB3 blockB4 boxB2 0.5 0.5 0 0

[0127] S740: Determine whether the overlap between the first text line detection box and multiple text block detection boxes is greater than the threshold.

[0128] The electronic device 100 determines whether the overlap between the first text line detection box and multiple text block detection boxes is greater than a threshold. If yes, that is, the overlap between the first text line detection box and multiple text block detection boxes is greater than the threshold, it means that the first text line detection box has detected the text of multiple columns as the same line, and then step S750 is executed. If no, that is, the overlap between the first text line detection box and only one text block detection box is greater than the threshold, it means that the first text line detection box contains only the text of one column, and then step S760 is executed.

[0129] The threshold value can be 0 or a small positive number. That is, when the first text line detection box and the text block detection box have a certain overlap, it can be considered that the first text line detection box has detected the text in the text block detection box.

[0130] S750: Divide the first text line detection box according to the coordinates of the text block detection box to obtain multiple second text line detection boxes.

[0131] When there is overlap between the first text line detection box and multiple text block detection boxes, the intersection points between the first text line detection box and multiple text block detection boxes will be determined first. The line connecting the two intersection points with the same text line detection box will be used as the dividing line to segment the first text line detection box, so that each of the multiple second text line detection boxes obtained by segmentation overlaps with only one text block detection box.

[0132] For example, such as Figure 9AAs shown, the electronic device 100 will determine the intersection points between boxB2 and blocksB1 and B2, where the intersection points between boxB2 and blockB1 are intersection points 1 and 4, and the intersection points between boxB2 and blockB2 are intersection points 2 and 3. Figure 9B As shown, the line connecting intersection points 1 and 4 is used as the dividing line, and the area to the left of this dividing line in boxB2 is defined as the left text line detection box boxB21 obtained by segmenting boxB2. Similarly, the line connecting intersection points 2 and 3 is used as the dividing line, and the area to the right of this dividing line in boxB2 is defined as the right text line detection box boxB22 obtained by segmenting boxB2. In this way, boxB21 obtained by segmenting boxB2 overlaps only with blockB1, and boxB22 obtained by segmenting boxB2 overlaps only with blockB2.

[0133] S760: Keep the first text line detection box unchanged.

[0134] If the overlap between the first text line detection box and only one text block detection box is greater than the threshold, it means that the first text line detection box contains only one column of text, and thus the first text line detection box remains unchanged.

[0135] In summary, by calculating the size of the overlapping area between the text line detection box and the text block detection box, it can be determined whether the text line detection box detects text from multiple columns as the same line. For cases where text from multiple columns is detected as the same line, the text line detection box is divided into multiple text line detection boxes, ensuring that the text in each text line detection box belongs to only one column. This avoids the problem of inconsistent text order between the recognized text and the original image caused by recognizing text from different columns as the same line, thus improving the accuracy and readability of the text recognition results.

[0136] To address the problem of misidentifying text in different columns as the same line in a multi-column layout, leading to incorrect recognition results, this application provides a text recognition device 1000, including an acquisition unit 1010 and a determination unit 1020.

[0137] The acquisition unit 1010 is used to acquire an image to be recognized containing text; the determination unit 1020 is used to determine a first text line detection box and multiple text block detection boxes in the image to be recognized; corresponding to the overlap between the first text line detection box and multiple text block detection boxes, the determination unit 1020 is also used to determine that the text in the first text line detection box belongs to multiple text block detection boxes; the first text line detection box is divided into multiple second text line detection boxes, wherein each second text line detection box overlaps with only one text block detection box.

[0138] In some embodiments, the determining unit 1020 is further configured to determine the size of the overlapping area between the first text line detection box and each text block detection box; corresponding to the size of the overlapping area between the first text line detection box and the multiple text block detection boxes being greater than a threshold, the determining unit 1020 is further configured to determine that the first text line detection box overlaps with the multiple text block detection boxes.

[0139] In other embodiments, corresponding to the fact that the size of the overlapping area between the first text line detection box and only one text block detection box is greater than a threshold, the determining unit 1020 is also used to determine that the text of the first text line detection box belongs to only one text block detection box.

[0140] In other embodiments, the determining unit 1020 is further configured to determine multiple intersection points between the first text line detection box and multiple overlapping text block detection boxes; the determining unit 1020 is further configured to divide the first text line detection box into multiple second text line detection boxes based on the multiple intersection points.

[0141] In other embodiments, the determining unit 1020 is further configured to obtain multiple dividing lines based on the line connecting two intersection points corresponding to the same text block detection box among multiple intersection points; and to divide the first text line detection box into multiple second text line detection boxes based on the multiple dividing lines.

[0142] In other embodiments, the determining unit 1020 is also used to perform text recognition based on multiple second text line detection boxes to obtain multiple line recognition results.

[0143] In other embodiments, the determining unit 1020 is further configured to determine the type of the image to be identified; corresponding to the image to be identified being a document image, the determining unit 1020 is further configured to determine the size of the overlapping area between the first text line detection box and each text block detection box, wherein the document image includes text in multiple columns.

[0144] In summary, the text recognition device 1000 provided in this application can determine whether a text line detection box recognizes text from multiple columns as the same line by calculating the size of the overlapping area between the text line detection box and the text block detection box. For cases where text from multiple columns is recognized as the same line, the text line detection box is divided into multiple text line detection boxes, ensuring that the text in each text line detection box belongs to only one column. This avoids the problem of inconsistent text order between the recognized text and the original image caused by recognizing text from different columns as the same line, thereby improving the accuracy and readability of the text recognition results.

[0145] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A character recognition method, characterized in that, The method includes: Obtain the image to be recognized that contains text; A first text line detection box and multiple text block detection boxes are determined in the image to be recognized. The first text line detection box corresponds to a line of text in the image to be recognized, and each text block detection box corresponds to a text block. If the first text line detection box overlaps with multiple text block detection boxes, it is determined that the text in the first text line detection box belongs to multiple text block detection boxes. The first text line detection box is divided into multiple second text line detection boxes, wherein each second text line detection box overlaps with only one text block detection box; The step of dividing the first text line detection box into multiple second text line detection boxes includes: Determine multiple intersection points between the first text line detection box and multiple overlapping text block detection boxes; Based on the multiple intersection points, the first text line detection box is divided into the multiple second text line detection boxes.

2. The character recognition method according to claim 1, characterized in that, The method further includes: Determine the size of the overlapping area between the first text line detection box and each text block detection box; If the size of the overlapping area between the first text line detection box and the multiple text block detection boxes is greater than the threshold, it is determined that the first text line detection box overlaps with the multiple text block detection boxes.

3. The character recognition method according to claim 2, characterized in that, The method further includes: If the size of the overlapping area between the first text line detection box and only one text block detection box is greater than a threshold, it is determined that the text in the first text line detection box belongs to only one text block detection box.

4. The character recognition method according to claim 2, characterized in that, The step of dividing the first text line detection box into the plurality of second text line detection boxes based on the plurality of intersection points includes: Multiple boundary lines are obtained by using the line connecting two intersection points corresponding to the same text block detection box among the multiple intersection points as the boundary lines; Based on the multiple dividing lines, the first text line detection box is divided into the multiple second text line detection boxes.

5. The character recognition method according to claim 4, characterized in that, The method further includes: Based on the multiple second text line detection boxes, text recognition is performed to obtain multiple line recognition results.

6. The character recognition method according to any one of claims 2 to 5, characterized in that, The method further includes: Determine the type of the image to be identified; Determining the size of the overlapping area between the first text line detection box and each text block detection box includes: Since the image to be identified belongs to the document type image, the size of the overlapping area between the first text line detection box and each text block detection box is determined, and the document type image includes text in multiple columns.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store instructions, the processor being used to execute the instructions, and when the processor executes the instructions, it performs the method as described in any one of claims 1 to 6.

8. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by an electronic device, enable the electronic device to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 6.

Images