Text line detection

Abstract

The implementation of the present disclosure provides a solution for text line detection. In the solution, a first text region including at least a first part of a first text element and a second text region including at least a second part of a second text element are determined from an image. A first feature representation is extracted from the first text region and a second feature representation is extracted from the second text region. The first feature representation and the second feature representation include at least one of an image feature representation of the image or a semantic feature representation of the image. A link relationship between the first text region and the second text region can then be determined based at least in part on the first feature representation and the second feature representation. The link relationship can indicate whether the first part and the second part of the first text element and the second text element are located in the same text line. In this way, by detecting text regions and determining their link relationships based on their feature representations, the accuracy and efficiency of detecting text lines in various images can be improved.

Description

Background Art

[0001] The technology of detecting and recognizing text lines in images is being widely used in many real-world applications such as text recognition, text translation and image tagging. Text lines in images refer to character sets and / or symbols that are spatially related or semantically related. For example, a user can capture an image with text using his / her smart phone and may expect the text lines in the image to be recognized. Compared with text line detection in well-captured document images (such as scanned versions of documents), text line detection in images of natural scenes is a more challenging task because of the huge diversity of both text appearance and surrounding background in the image. For example, text lines in natural scene images may be in any orientation and may vary greatly in font, size, color, etc. Therefore, it is necessary to improve the accuracy and efficiency of text line detection in images. Summary of the invention

[0002] According to an implementation of the subject matter described herein, a solution for text line detection is provided. In the solution, a first text region including at least a first portion of a first text element and a second text region including at least a second portion of a second text element are determined from an image. A first feature representation is extracted from the first text region and a second feature representation is extracted from the second text region. The first feature representation and the second feature representation include at least one of an image feature representation or a semantic feature representation of text content contained in the image. A link relationship between the first text region and the second text region can then be determined based at least in part on the first feature representation and the second feature representation. The link relationship can indicate whether the first portion and the second portion of the first text element and the second text element are located in the same text line. In this way, by detecting text regions and determining their link relationships based on their feature representations, the accuracy and efficiency of detecting text lines in various images can be improved.

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 shows a block diagram of a computing device in which various implementations of the subject matter described herein may be implemented;

[0005] Figure 2 A flowchart showing a process for text line detection according to an implementation of the subject matter described herein;

[0006] Figure 3shows an example structure of a text region detection module according to an implementation of the subject matter described herein;

[0007] Figure 4 shows an example structure of a link relationship determination module according to an implementation of the subject matter described herein;

[0008] Figure 5 shows an example structure for text line region determination according to implementations of the subject matter described herein; and

[0009] Figure 6 A flow chart illustrating a process for providing a text line region according to an implementation of the subject matter described herein.

[0010] Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. DETAILED DESCRIPTION

[0011] The subject matter described herein will now be discussed with reference to a number of example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement the subject matter described herein, and do not imply any limitation on the scope of the subject matter.

[0012] As used herein, the term "including" and variations thereof should be understood as open terms meaning "including but not limited to." The term "based on" should be understood as "based at least in part on." The terms "an implementation" and "implementation" should be understood as "at least one implementation." The term "another implementation" should be understood as "at least one other implementation." The terms "first," "second," etc. may refer to different or the same objects. Other definitions (whether explicit or implicit) may be included below.

[0013] Example Environment

[0014] Text line detection plays an important role in many content-based visual intelligence applications, such as image retrieval, autonomous driving, OCR translation, etc. Conventional text line detection techniques can be well applied to capture good document images. However, due to the huge diversity of both text appearance and surrounding background in these images, these conventional text line detection methods perform poorly in images of natural scenes. A good text line detection solution should be able to robustly detect text lines of any language and any orientation from both scanned document images and natural scene images. However, due to the huge diversity of both text appearance and surrounding background, existing text detection methods (including both top-down and bottom-up methods) cannot achieve this goal. In the implementation of the subject described in this article, a solution for general text line detection is provided.

[0015] Figure 11 is a block diagram of a computing device 100 in which various implementations of the subject matter described herein may be implemented. It should be understood that Figure 1 The computing device 100 shown is for illustrative purposes only and is not intended to imply any limitation in any way on the functionality and scope of implementations of the subject matter described herein. Figure 1 As shown, computing device 100 comprises a general computing device 100. Components of computing device 100 may include, but are not limited to, one or more processors or processing units 110, memory 120, storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160.

[0016] In some implementations, the computing device 100 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server, a large computing device, etc. provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 100 can support any type of interface to the user (such as, a "wearable" circuit system, etc.).

[0017] Processing unit 110 may be a physical or virtual processor and may perform various processes based on a program stored in memory 120. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 100. Processing unit 110 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0018] The computing device 100 typically includes various computer storage media. Such media can be any media accessible to the computing device 100, including but not limited to volatile and non-volatile media, or removable and non-removable media. The memory 120 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory), or any combination thereof. The storage device 130 can be any removable or non-removable media, and can include machine-readable media, such as a memory, a flash drive, a disk, or other media, which can be used to store information and / or data and can be accessed in the computing device 100.

[0019] The computing device 100 may also include additional removable / non-removable volatile / non-volatile storage media. Figure 1 Not shown in the figure, but a magnetic disk drive for reading and / or writing a removable non-volatile magnetic disk and an optical disk drive for reading and / or writing a removable non-volatile optical disk can be provided. In this case, each drive can be connected to a bus (not shown) via one or more data medium interfaces.

[0020] The communication unit 140 communicates with another computing device via a communication medium. In addition, the functionality of the components in the computing device 100 may be implemented by a single computing cluster or multiple computing machines that may communicate via a communication connection. Thus, the computing device 100 may operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0021] Input device 150 may be one or more of a variety of input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 160 may be one or more of a variety of output devices, such as a display, speaker, printer, etc. If necessary, through communication unit 140, computing device 100 may further communicate with one or more external devices (not shown) (such as storage devices and display devices), with one or more devices that enable a user to interact with computing device 100, or with any device that enables computing device 100 to communicate with one or more other computing devices (such as a network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0022] In some implementations, as an alternative to being integrated in a single device, some or all components of the computing device 100 may also be arranged in a cloud computing architecture. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the subject matter described herein. In some implementations, cloud computing provides computing, software, data access, and storage services, which will not require end users to understand the physical location or configuration of the system or hardware that provides these services. In various implementations, cloud computing provides services via a wide area network (such as the Internet) using a suitable protocol. For example, a cloud computing provider provides an application via a wide area network, which can be accessed by a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a server at a remote location. The computing resources in a cloud computing environment may be merged or distributed at locations in a remote data center. Cloud computing infrastructure may provide services through a shared data center, although they act as a single access point for users. Therefore, a cloud computing architecture may be used to provide the components and functions described herein from a service provider at a remote location. Alternatively, they may be provided from a conventional server or installed directly or otherwise on a client device.

[0023] Computing device 100 may be used to implement text line detection in implementations of the subject matter described herein. Therefore, hereinafter, computing device 100 is also referred to as a "text line detection device". Memory 120 may include one or more text line detection modules 122 having one or more program instructions. These modules are accessible and executable by processing unit 110 to perform the functions of various implementations described herein.

[0024] To implement text line detection, computing device 100 may receive image 170 via input device 150. Text line detection module 122 performs text line detection on image 170 and may determine one or more text line regions 185. Text line detection module 122 may further determine the coordinates of the bounding boxes of text line regions 185 in image 170. In some embodiments, output device 160 may output a file including the bounding box coordinates of each of the detected text line regions 185. In some embodiments, for visualization purposes, text line detection module 122 may further generate an updated image 180 in which the bounding boxes of the detected text line regions 185 are highlighted to indicate that the text elements including text line regions 185 are located in the same text line. Output device 160 may present, for example, updated image 180 to the user.

[0025] As used herein, a "text element" refers to an element used to present text or graphic information. Examples of "text elements" may include, but are not limited to, characters or symbols, such as English letters, Chinese characters, punctuation marks, or graphic marks, etc. In some cases, a "text element" may also be referred to as a "text instance."

[0026] As used herein, a "text region" refers to a region of an image that is detected as including at least a portion of at least a text element. In some cases, a "text region" may also be referred to as a "text primitive". For example, a text region may include half of a text element, a complete text element, multiple complete text elements, or multiple portions of different text elements. As used herein, a "text line" refers to a set of text elements that form a line in an image according to their spatial and / or semantic relationships. As used herein, a "text line region" refers to a region of an image that defines at least a portion of the group of text elements that are determined to be in the same text line.

[0027] It should be understood that Figure 1 The images and text line regions shown are for illustration purposes only. In other examples, any image can be processed and any text line region can be determined accordingly.

[0028] Roughly speaking, there are two types of conventional deep learning-based text line detection solutions, namely, top-down methods and bottom-up methods. Top-down methods are not robust to long slanted / curved text lines in the vicinity. Bottom-up methods either predict a pixel-level text score map or first detect candidate text segments, and then use different methods to group the detected text segments or text pixels into text lines and calculate the corresponding bounding boxes. Pixel-based methods cannot robustly detect text lines with large inter-character distances. The line grouping method used by text segmentation-based methods is not robust, so these methods are difficult to apply to images in which the text elements included are organized in an irregular manner. (Here, the limitations of segmentation-based methods and pixel-based methods are different.)

[0029] To implement the working principle and example structure of text line detection

[0030] According to implementation of the subject matter described herein, a solution for text line detection is proposed. In the solution, text regions including at least a portion of text elements are detected. A link relationship between the text region pairs is further determined based on feature representations extracted from the text region pairs. The link relationship may indicate whether the text elements included in the pair of text regions are located in the same text line. In this way, by detecting text regions and determining their link relationships based on their feature representations, the accuracy and efficiency of detecting text lines in various images may be improved.

[0031] The general principles and several example implementations of the subject matter described herein are described below with reference to the accompanying drawings.

[0032] First reference Figure 2 , Figure 2 A block diagram of a text line detection module according to an implementation of the subject matter described herein is shown. For illustration purposes, Figure 1 The text line detection module 122 in the computing device 100 is referred to as an example for implementing the text line detection described herein. The text line detection module 122 includes multiple modules for implementing multiple stages of detecting one or more text lines in an image.

[0033] The text line detection process can be done using Figure 2 As shown in the figure, the text line detection module 122 includes a text area detection stage or module 210 and a link relationship determination stage or module 220. In some implementations, the text line detection module 122 may also include a text line area determination stage or module 230.

[0034] In some implementations, image 170 may be provided as input to text region detection module 210. Text region detection module 210 may determine a plurality of text regions 212 in image 170, for example by using machine learning techniques. Each determined text region 212 in image 170 may include at least a portion of a text element.

[0035] For example, the text region detection module 210 may first determine multiple candidate regions based on a sliding window, and then obtain a feature map of each candidate region in the multiple candidate regions. The feature map may then be input into a machine learning model to determine whether the corresponding candidate region is a text region. It should be understood that the specific text regions discussed above are for illustrative purposes only. Any method currently used or to be developed in the future for automatic or manual assisted text region detection may be used to detect text regions in the image 170.

[0036] As multiple text regions 212 are determined, the link relationship determination module 220 is configured to detect link relationships 222 between pairs of text regions in the multiple text regions 212. The link relationship 222 may indicate whether the (multiple) text elements or (multiple) parts of (multiple) text elements included in the text region pair are located in the same text line. In some implementations, the link relationship determination module 220 may first extract corresponding feature representations from the text region pair, respectively. The link relationship 222 between the text region pairs may then be determined based on the extracted feature representations using a machine learning model. The determination of the link relationship based on the extracted feature representations will be discussed in detail below.

[0037] In some implementations, the text line detection module 122 may further include a text line region determination stage or module 230. The results of the link relationship determination module 220 (i.e., the link relationship between the text regions) are provided to the text line region determination stage or module 230. The text line region determination stage or module 230 is configured to determine which text regions 212 are located in the same text line based on the link relationship, and then determine the text line region 185 that bounds these text regions in the image 170. The process in the text line region determination 230 will be discussed in detail below.

[0038] In some implementations, the result of the text line region determination module 230 (i.e., the determined text line region(s) 185) may be further used for other purposes, such as for text recognition, mobile search, text translation, etc. For example, the text line region 185 may be provided to an image-based text recognition module for identifying specific text presented in the text line region 185. Alternatively or additionally, an updated image 180 may be generated in which the bounding boxes of the determined text line region(s) are highlighted. For example, by clearly presenting the determined text line region to, for example, a developer, it may be clearly shown whether the text line detected by the text line detection module 230 is accurate.

[0039] It should be understood that Figure 2 The modules in the can be implemented as one or more software engines, hardware components, or a combination thereof, etc., which are configured with logic for implementing the functions attributed to the specific module. Each module can be implemented using one or more of such software engines, hardware components, etc. The software engines, etc. are executed on one or more processors of one or more computing systems or devices, and utilize or operate data stored in one or more storage devices, memories, etc. on one or more computing systems. In some implementations, Figure 2 The different modules in can be implemented as a single module, and Figure 2 A single module in the text line detection module 122 may be separated into more than one module. In some implementations, one or more additional modules may be included in the text line detection module 122.

[0040] Text Region Detection

[0041] To detect one or more text regions in image 170, text region detection module 122 may determine multiple candidate regions, for example, based on a sliding window-based approach. In some implementations, to detect text elements of various sizes, text region detection module 122 may use sliding windows of different sizes to detect candidate regions of corresponding sizes from image 170.

[0042] Figure 3An example stricter 300 of the text region detection module 210 according to an implementation of the subject matter described herein is shown. Figure 3 In the example of , a feature pyramid network (FPN) is used for feature extraction. Specifically, image 170 is provided as an input to FPN 310 for generating feature maps with different sizes. FPN 310 has two paths, namely a bottom-up path 315 and a top-down path 320. The bottom-up path 315 is a feedforward calculation of a backbone convolutional neural network (ConvNet), which calculates a feature hierarchy consisting of feature maps of several scales with a scale step of 2. Figure 3 In the example of , a 4-level feature pyramid is included, with strides of 4, 8, 16, and 32 pixels, respectively. The top-down path 320 generates higher resolution features by upsampling spatially coarser but semantically stronger feature maps from higher pyramid levels. These features are then enhanced using the features of the bottom-up path via lateral connections. It should be appreciated that specific parameters related to FPN (e.g., number of levels, stride, etc.) are for illustrative purposes only, and any appropriate model parameters may be applied.

[0043] like Figure 3 As shown, three scale-specific text region prediction modules 330-1, 330-2, and 330-3 (individually or collectively referred to herein as text region prediction modules 330) are respectively configured to receive feature maps 325-1, 325-2, and 325-3 (individually or collectively referred to herein as feature maps 325) and are respectively designed to detect small (e.g., 4px-23px), medium (e.g., 24px-48px), and large (e.g., >48px) text elements. In some implementations, a region proposal network (RPN) can be applied to each of the text region prediction modules 330-1, 330-2, and 330-3. For example, the text region prediction modules 330-1, 330-2, and 330-3 can apply an anchor-free RPN to perform dense pixel-by-pixel prediction of text elements.

[0044] For example, if a pixel of a pyramid level is located in the core area of ​​a text element of that pyramid level, the corresponding text region prediction module 330 may mark the pixel as a "text" pixel and directly predict the offsets from it to the vertices of its corresponding quadrilateral text region. In some implementations, each module in the text region prediction module 330 may be implemented as a 3×3 convolutional layer followed by two sibling 1×1 convolutional layers for text / non-text classification and quadrilateral bounding box regression, respectively.

[0045] In some implementations, the text region prediction module 330 may first generate a score (e.g., 0 to 1) for each candidate region in the candidate area, where the score may represent the probability that the corresponding candidate region is a text region. The text region prediction module 330 may then determine the corresponding text region 212 in the image 170 by comparing the score of each candidate region with a score threshold (e.g., 0.85). In this way, candidate regions with scores higher than the threshold will be determined as text regions. For example, Figure 3 As shown, two small text regions 212-1 and 212-6 may be determined by the text region prediction module 330-1, a medium text region 212-2 may be determined by the text region prediction module 330-2, and three text regions 212-3, 212-4, and 212-5 may be determined by the text region prediction module 330-3.

[0046] like Figure 3 As shown, text area 212-1 includes a relatively small-sized character "M", and text area 212-3 includes a relatively large-sized character "R". It should be understood that Figure 3 For the purpose of illustration, text regions 312-1, 312-2, 312-3, 312-4, 312-5, and 312-6 are shown in FIG. 1 , and there may be text regions 312-1, 312-2, 312-3, 312-4, 312-5, and 312-6 determined by the text region prediction module 330. Figure 3 Other text areas 212 are not shown. Figure 3 The text area shown includes only complete characters, but it should be understood that the text area may also include parts of characters, symbols, parts of symbols, multiple characters, multiple symbols, a combination of (multiple) characters and (multiple) symbols, etc.

[0047] It should be noted that the FPN 310 and the RPN included in the text region prediction module 330 can be trained together to determine text regions of different sizes in an image. The total loss in the joint training can be determined by calculating the sum of the losses of the models applied in the scale-specific text region prediction module 330.

[0048] The specific structure of the text region detection module 210 as discussed above is merely exemplary and is not intended to limit the scope of the subject matter. Any other suitable structure for the text region detection module may be employed to detect text regions in the image 170.

[0049] Link relationship determination

[0050] Using the text region 212 determined by the text region detection module 122, the link relationship determination module 220 may first select a first text region and a second text region from a plurality of text regions. Figure 3For example, the link relationship determination module 220 may traverse all text regions 212 determined by the text region detection module 122 .

[0051] In some implementations, the link relationship determination module 220 may determine a link relationship between a text region 212 and another text region 212. Figure 4 Discuss the process of link relationship determination. Figure 4 An example structure 400 of the link relationship determination module 220 according to an implementation of the subject matter described herein is shown.

[0052] As shown in the figure, the link relationship determination module 220 includes a feature extraction module 410 configured to extract feature representations from text region pairs and a relationship prediction model 440 configured to determine the link relationship between the text region pairs.

[0053] by Figure 4 The text region 212 - 3 is shown as an example of a first text region. In some implementations, the link relationship determination module 220 may determine a link relationship between the text region 212 - 3 and each of the other text regions.

[0054] Alternatively, considering that the sizes of text elements in the same text line should be similar, the feature extraction module 410 can select the second text region from the other text regions by determining whether the difference between the first size of the text region 212-3 and the size of another text region is below a size threshold. When it is determined that the size difference is below the size threshold, the feature extraction module 410 can continue to extract feature representations from the text region pair. Otherwise, the feature extraction module 410 can ignore the text region pair.

[0055] For example, for Figure 4 For text region 212-3 in the figure, feature extraction module 410 may determine that the size difference between text regions 212-2 and 212-3 exceeds the size threshold. Therefore, feature extraction module 410 may ignore the text region pair 212-2 and 212-3. Figure 4 As shown, the size of the text region 212-4 is similar to the size of the text region 212-3, so the feature extraction module 410 can extract feature representations of the two text regions. In this way, the efficiency of determining the link relationship can be improved by reducing unnecessary calculations.

[0056] Alternatively or additionally, considering that the distance between two adjacent text regions should not be too large, the feature extraction module 410 can select the second text region from the other text regions by determining whether the first distance between the first center of the text region 212-3 and the center of the other region is below a distance threshold. In some implementations, the distance threshold can be set based on the size of the first text region, for example, five times the height of the first text region.

[0057] When it is determined that the distance is lower than the distance threshold, the feature extraction module 410 may continue to extract feature representations from the pair of text regions. Otherwise, the pair of text regions may be ignored accordingly.

[0058] For example, about Figure 4 For text region 212-3 in the text region 212-5, feature extraction module 410 may determine that the distance between the first center of text region 212-3 and the second center of text region 212-5 exceeds the distance threshold. Therefore, feature extraction module 410 may ignore the pair of text regions 212-2 and 212-5. Figure 4 As shown, the distance between the centers of the text regions 212 - 3 and 212 - 4 is less than the distance threshold, and thus feature representations of the two text regions may be extracted by the feature extraction module 410 .

[0059] In some implementations, the feature extraction module 410 can consider both size difference and distance. In an example of using FPN to generate feature maps of different scales, such as Figure 3 As shown, the feature extraction module 410 may only consider pairs of text regions detected from the same pyramid level and whose centers are at a distance less than a distance threshold. Figure 4 As shown in the example of , the feature extraction module 410 will consider two text region pairs (text regions 212 - 3 and 212 - 4 , and text regions 212 - 4 and 212 - 5 ).

[0060] In the case where the first text region and the second text region are selected, the feature extraction module 410 can then extract a first feature representation from the first text region and a second feature representation from the second text region. In some implementations, the first feature representation and the second feature representation may include image feature representations. For example, the feature extraction module 410 can extract a first feature representation from the first text region and a second feature representation from the second text region. Figure 3 The first feature representation and the second feature representation are obtained from the feature map 325 generated by the FPN 310 shown.

[0061] Alternatively, the first feature representation and the second feature representation may include semantic feature representations. The feature extraction module 410 may first use any appropriate image-based text recognition technology to identify at least the first text element and the second text element included in the first area and the second area. The feature extraction module 410 may then generate a first semantic feature representation and a second semantic feature representation based on the first text element and the second text element identified therein, for example by using word2vec technology. It should be understood that any method for text semantic feature representation generation (whether existing or to be developed in the future) can be used to generate semantic feature representations of the identified representations. By using semantic feature representations, the solution implemented according to the present disclosure can accurately detect text lines in which text elements are not very related spatially but are very related semantically.

[0062] like Figure 4 As shown in the example of , the feature extraction module 410 can extract the first feature representation 420 from the first text region 212-3 and the second feature representation 425 from the second text region 212-4. In some implementations, the feature extraction module 410 can then combine the first feature representation and the second feature representation into a combined feature representation 435.

[0063] Then, the combined feature representation 435 may be provided to the relationship prediction model 440. The relationship prediction model 440 is configured to determine the link relationship 222 between the first text region and the second text region based on the combined feature representation 435. The relationship prediction model 440 may be any suitable type of machine learning model, examples of which may include a gradient boosted decision tree (GBDT) model, a support vector machine (SVM), a random forest, a neural network, and the like.

[0064] During the training of the relationship prediction model 440, multiple positive training samples and negative training samples may be applied. In some implementations, positive training samples may be generated by selecting a pair of text regions located in a known text line identified, for example, by a user. Negative training samples may be generated by selecting a pair of text regions identified as being located in two different known text lines. During training, feature representations may be extracted from the text region pairs included in the training samples and then input into the relationship prediction model. The parameters of the relationship prediction model may be adjusted round by round to make the prediction results close to the true results of the training samples.

[0065] In some implementations, the feature extraction module 410 may further extract the third feature representation 430 based on both the first text region and the second text region. For example, the link relationship determination module 220 may extract the third feature representation 430 by determining the spatial relationship between the first text region and the second text region. In some implementations, the feature extraction module 410 may first determine a first coordinate set associated with the first text region and a second coordinate set associated with the second text region. For example, the first coordinate set and the second coordinate set may include corresponding coordinates of bounding boxes of the first region and the second region in the image 170.

[0066] The feature extraction module 410 can further determine a third feature representation 430 based on the first set of coordinates and the second set of coordinates. For example, the third feature representation can be an 18-dimensional vector of concatenated 6-D vectors, each vector indicating a difference between the first text region and the second text region. For example, the 6-D vector can include where each dimension is given by:

[0067]

[0068]

[0069]

[0070] where x S and S represents the coordinates of the center of the first text area, w S and h S Indicates the width and height of the first text area, x O and O represents the coordinates of the center of the second text area, w O and h O Indicates the width and height of the second text area.

[0071] Alternatively, the third feature representation 430 may also be determined by extracting features from a third text region that at least delimits the first text region and the second text region. Figure 4 In the example shown, the feature extraction module 410 may first determine the third text region 415 based on the first text region 212-3 and the second text region 212-4. Figure 4 As shown, the third text region 415 bounds both the first text region 212 - 3 and the second text region 212 - 4 .

[0072] In addition, the feature extraction module 410 can then extract a third representation 430 from the determined third text region 415. In some implementations, the third feature representation 430 can be an image feature representation extracted from the third text region 415 in the image. Alternatively, the third feature representation 430 can be a semantic feature representation extracted from the third text region 415. The process of extracting the third feature representation 430 can be performed with reference to the process of extracting the first feature representation 420 and the second feature representation 425 from the first text region and the second text region as described above, which will not be discussed in detail here.

[0073] like Figure 4 As shown, in the example of extracting the third feature representation 430, the extracted first feature representation 420, the second feature representation 425, and the third feature representation 430 can be further cascaded together to form a combined feature representation 435, and then provided to the relationship prediction model 440 for determining the link relationship 222 between the first text region 212-3 and the second text region 212-4. It should be understood that during the training of the relationship prediction model 440, the input of the model should also include the third feature representation, which is generated from the text region pairs included in the training sample. By considering the additional third feature representation 430 during the prediction of the relationship prediction model 440, a more accurate prediction result can be determined.

[0074] By using the feature representation extracted from the text region to predict the link relationship through the above method, the solution implemented according to the present disclosure can improve the accuracy of relationship determination. For example, compared with the conventional text line detection solution based on image segmentation, the solution implemented according to the present disclosure can accurately determine the text lines where two text regions are located at a relatively large distance.

[0075] Furthermore, in some implementations, both image and semantic feature representations may be considered during link relationship determination. This will help determine the link relationship of two text regions that are semantically related, even though the text regions are not very spatially related. For example, a user may sign his / her name after printing the word "name" on paper, and the signed name may be far away from the printed word "name" and not completely aligned with the word "name". Incorporating semantic feature representations will help identify that the signed name and the word "name" are in the same line.

[0076] Text line area determination

[0077] The link relationship 222 between each text region pair 212 included in the image may be further provided to a text line region determination module 230. The text line region determination module 230 is configured to determine the text line region 185 based on the link relationship 222. Figure 5Discuss the process of determining text line regions. Figure 5 An example structure 500 of a text line region determination module 230 is shown according to an implementation of the subject matter described herein.

[0078] As shown in the figure, the text line region determination module 230 includes a text region grouping module 510 and a text line region disambiguation module 520. The text region grouping module 510 is configured to group the text regions 212 into different sets based on the link relationship 222 to determine one or more candidate text line regions, and the text line region disambiguation module 520 is configured to disambiguate multiple overlapping candidate text line regions.

[0079] In some implementations, the text region grouping module 510 can receive the link relationship 222 and group the text regions 212 into different sets accordingly. For example, if the link relationship indicates that the text regions 212-3 and 212-4 are related to the same text line and the text regions 212-4 and 212-5 are related to the same text line, the text region grouping module 510 can group the text regions 212-3, 212-4, and 212-5 into the same set.

[0080] The text region grouping module 510 can then determine the candidate text line region 185 based on the grouped text region set. In some implementations, the text region grouping module 510 can determine the candidate text line region that at least defines the text regions in the image that are grouped into the same text region set. For example, the text region grouping module 510 can determine the candidate text line region 515-1 based at least in part on the coordinates of the bounding boxes of the text regions 212-3, 212-4, and 212-5, and the coordinates of other text regions grouped into the same text region set as them.

[0081] The determined candidate text line region(s) are provided to the text line region disambiguation module 520. In some implementations, multiple scale-specific text region prediction modules (e.g., scale-specific text region prediction module 330) are used, and thus multiple overlapping candidate text line regions may be determined for the same text line. Figure 5 As shown, two candidate text line regions 515 - 1 and 515 - 2 (individually or collectively referred to as candidate text line regions 515 ) are determined by the text line grouping module 510 . To avoid generating multiple overlapping text line regions for the same text line, the text line region disambiguation module 520 can determine the final text line region from the candidate text line regions 515 .

[0082] For example, for the candidate text line region 515-1, the text line region disambiguation module 520 may further select a candidate text line region 515-2 in the image 170. In some embodiments, the text line region disambiguation module 520 may select the candidate text line region 515-2 based on determining that the overlap area between the candidate text line regions 515-1 and 515-2 exceeds an area threshold. Alternatively, the text line region disambiguation module 520 may select the candidate text line region 515-2 based on determining that the ratio of the overlap area between the candidate text line regions 515-1 and 515-2 to the union area of ​​the candidate text line regions 515-1 and 515-2 exceeds a ratio threshold.

[0083] In some embodiments, the text line region disambiguation module 520 can disambiguate the two candidate text line regions 515-1 and 515-2 based on their corresponding scores. As described above, a score can be assigned to each text region by the text region prediction module 330 to indicate the possibility that the text region includes at least a portion of a text element. The score of the candidate text line region can be determined as the score of the text region included in the candidate text line region. In addition, the text line region disambiguation module 520 can only provide higher scores to the candidate text line regions.

[0084] Alternatively, the text line region disambiguation module 520 may apply another relationship prediction model to disambiguate the two candidate text line regions 515-1 and 515-2. Figure 3 The text line region disambiguation module 520 can extract feature representations from the two candidate text line regions 515-1 and 515-2, respectively. It should be understood that the extraction process of the fourth feature representation and the fifth feature representation can be performed with reference to the extraction technology of the first feature representation and the second feature representation, which will not be repeated here.

[0085] In some implementations, the text line region disambiguation module 520 can further determine a sixth feature representation based on both the text line regions 515-1 and 515-2. The process of determining the sixth feature representation can be the same as described above with reference to Figure 3 The process 430 of determining the third feature representation discussed above is performed together. Figure 3 In the process of , the combination of the fourth feature representation, the fifth feature representation, and the sixth feature representation can be provided to the relationship prediction model included in the text line region disambiguation module 520 to determine which region of the candidate text line regions 515-1 and 515-2 to provide. The relationship prediction model included in the text line region disambiguation module 520 can be any appropriate type of machine learning model, examples of which can include a gradient boosted decision tree (GBDT) model, a support vector machine (SVM), a random forest, a neural network, etc.

[0086] For the training of the relationship prediction model included in the text line region disambiguation module 520, training samples can be generated by selecting text line region pairs, where the overlapping area between the text line region pairs exceeds the region threshold, and the ground-truth used for training can be the user's selection of a more accurate text line region.

[0087] In some implementations, based on determining to provide a candidate text line region 515-1, the text line region disambiguation module 520 may only provide the text line region 185-3 (the candidate text line region 515-1) in the image without providing the candidate text line region 515-2. In this way, the solution implemented according to the present disclosure can provide a more accurate text line region, thereby avoiding providing duplicate text line regions for the same text line.

[0088] In some embodiments, the text line region disambiguation module 520 may generate a file including coordinates of a bounding box for each text line region in the determined text line regions 185. In some implementations, the text line region disambiguation module 520 may also generate an updated image 180 in which the text line region 515-1 is highlighted. In this way, the determined text line region 185 may be clearly presented to, for example, a user, for confirmation of whether the text line detection is accurate.

[0089] Example Process

[0090] Figure 6 Flowchart of process 600 for text line detection according to some implementations of the subject matter described herein is shown. Process 600 may be implemented by computing device 100, for example, by text line detection module 122 in computing device 110. Process 600 may also be implemented by any other device or device cluster similar to computing device 100. For the purpose of description, reference is made to Figure 1 Process 600 is described.

[0091] At box 610, computing device 100 determines a first text region and a second text region in an image, wherein the first text region includes at least a first portion of a first text element and the second text region includes at least a second portion of a second text element. At box 620, computing device 100 extracts a first feature representation from the first text region and a second feature representation from the second text region, wherein the first feature representation and the second feature representation include at least one of an image feature representation or a semantic feature representation. At box 630, computing device 100 determines a link relationship between the first text region and the second text region based at least in part on the first feature representation and the second feature representation, wherein the link relationship indicates whether the first portion and the second portion of the first text element and the second text element are located in the same text line.

[0092] In some implementations, determining the first text region and the second text region includes: identifying multiple text regions from an image; selecting the first text region and the second text region from the multiple text regions based on at least one of: determining that a difference between a first size of the first text region and a second size of the second text region is below a size threshold, or determining that a first distance between a first center of the first text region and a second center of the second text region is below a distance threshold.

[0093] In some implementations, determining the link relationship between the first text region and the second text region includes: extracting a third feature representation indicating at least one of the following: a spatial relationship between the first text region and the second text region, or a semantic relationship between the first text element and the second text element; and further determining the link relationship based on the third feature representation.

[0094] In some implementations, extracting the third feature representation includes: determining a third text region that bounds at least the first text region and the second text region; and extracting the third feature representation from the third text region.

[0095] In some implementations, determining the third feature representation includes: obtaining a first set of coordinates associated with the first text region and a second set of coordinates associated with the second text region; and determining the third feature representation based on the first set of coordinates and the second set of coordinates.

[0096] In some implementations, the method further includes: determining a first text line region that at least defines the first text region and the second text region in the image based on a link relationship indicating that at least a portion of the text elements in the first text region and at least a portion of the text elements in the second text region are located in the same text line; and providing the first text line region.

[0097] In some implementations, providing a first text line region includes: determining a second text line region in an image, an overlapping area between the first text line region and the second text line region exceeding an area threshold; extracting a fourth feature representation from the first text line region and extracting a fifth feature representation from the second text line region; determining whether the first text line region is to be provided using another machine learning model and based at least in part on the fourth feature representation and the fifth feature representation; and providing the first text line region in the image without providing the second text line region based on determining that the first text line region is to be provided.

[0098] In some implementations, providing the first text line region includes: highlighting the first text line region in the image.

[0099] In some implementations, the first text element includes at least one of a character or a symbol.

[0100] Example Implementation

[0101] Listed below are some example implementations of the topics described in this article.

[0102] In a first aspect, the subject matter described herein provides a computer-implemented method. The method includes determining a first text region and a second text region in an image, the first text region including at least a first portion of a first text element, and the second text region including at least a second portion of a second text element; extracting a first feature representation from the first text region and extracting a second feature representation from the second text region, the first feature representation and the second feature representation including at least one of an image feature representation or a semantic feature representation; and determining a link relationship between the first text region and the second text region using a machine learning model and based at least in part on the first feature representation and the second feature representation, the link relationship indicating whether the first portion of the first text element and the second portion of the second text element are located in the same text line.

[0103] In some embodiments, extracting a first feature representation from a first text region and extracting a second feature representation from a second text region includes: identifying at least a first text element from the first text region and identifying at least a second text element from the second text region; and generating a first semantic feature representation based on the identified first text element and generating a second semantic feature representation based on the identified second text element.

[0104] In some implementations, determining the first text region and the second text region includes: identifying multiple text regions from an image; and selecting the first text region and the second text region from the multiple text regions based on at least one of: determining that a difference between a first size of the first text region and a second size of the second text region is below a size threshold, or determining that a first distance between a first center of the first text region and a second center of the second text region is below a distance threshold.

[0105] In some implementations, determining the link relationship between the first text region and the second text region includes: extracting a third feature representation indicating at least one of the following: a spatial relationship between the first text region and the second text region, or a semantic relationship between the first text element and the second text element; and further determining the link relationship based on the third feature representation.

[0106] In some implementations, extracting the third feature representation includes: determining a third text region that bounds at least the first text region and the second text region; and extracting the third feature representation from the third text region.

[0107] In some implementations, determining the relationship feature set includes: obtaining a first coordinate set associated with the first text region and a second coordinate set associated with the second text region; and determining a third feature representation based on the first coordinate set and the second coordinate set.

[0108] In some implementations, the method further includes: determining a first text line region that defines at least the first text region and the second text region in the image based on a link relationship indicating that at least a portion of a first text element in the first text region and at least a portion of a second text element in the second text region are located in the same text line; and providing the first text line region.

[0109] In some implementations, providing a first text line region includes: determining a second text line region in an image, an overlapping area between the first text line region and the second text line region exceeding an area threshold; extracting a fourth feature representation from the first text line region and extracting a fifth feature representation from the second text line region; determining whether the first text line region is to be provided using another machine learning model and based at least in part on the fourth feature representation and the fifth feature representation; and providing the first text line region in the image without providing the second text line region based on determining that the first text line region is to be provided.

[0110] In some implementations, providing the first text line region includes: highlighting the first text line region in the image.

[0111] In some implementations, the first text element includes at least one of a character or a symbol.

[0112] In a second aspect, the subject matter described herein provides an electronic device. The electronic device includes a processing unit; and a memory coupled to the processing unit and having instructions stored thereon, the instructions, when executed by the processing unit, causing the electronic device to perform an action, the action comprising: determining a first text region and a second text region in an image, the first text region including at least a first portion of a first text element, and the second text region including at least a second portion of a second text element; extracting a first feature representation from the first text region and extracting a second feature representation from the second text region, the first feature representation and the second feature representation including at least one of an image feature representation or a semantic feature representation; and determining a link relationship between the first text region and the second text region based at least in part on the first feature representation and the second feature representation, the link relationship indicating whether the first portion of the first text element and the second portion of the second text element are located in the same text line.

[0113] In some implementations, extracting a first feature representation from a first text region and extracting a second feature representation from a second text region includes: identifying at least a first text element from the first text region and identifying at least a second text element from the second text region; and generating a first semantic feature representation based on the identified first text element and generating a second semantic feature representation based on the identified second text element.

[0114] In some implementations, determining the link relationship between the first text region and the second text region includes: extracting a third feature representation indicating at least one of the following: a spatial relationship between the first text region and the second text region, or a semantic relationship between the first text element and the second text element; and further determining the link relationship based on the third feature representation.

[0115] In some implementations, extracting the third feature representation includes: determining a third text region that at least bounds the first text region and the second text region; and extracting the third feature representation from the third text region.

[0116] In some implementations, determining the relationship feature set includes: obtaining a first coordinate set associated with the first text region and a second coordinate set associated with the second text region; and determining a third feature representation based on the first coordinate set and the second coordinate set.

[0117] In some implementations, the action also includes: determining a first text line region that defines at least the first text region and the second text region in the image based on a link relationship indicating that at least a portion of a first text element in the first text region and at least a portion of a second text element in the second text region are located in the same text line; and providing the first text line region.

[0118] In some implementations, providing a first text line region includes: determining a second text line region in an image, an overlapping area between the first text line region and the second text line region exceeding an area threshold; extracting a fourth feature representation from the first text line region and extracting a fifth feature representation from the second text line region; determining whether the first text line region is to be provided using another machine learning model and based at least in part on the fourth feature representation and the fifth feature representation; and providing the first text line region in the image without providing the second text line region based on determining that the first text line region is to be provided.

[0119] In some implementations, providing the first text line region includes: highlighting the first text line region in the image.

[0120] In some implementations, the first text element includes at least one of a character or a symbol.

[0121] In a third aspect, the subject matter described herein provides a computer program product tangibly stored on a computer storage medium and comprising machine executable instructions which, when executed by a device, cause the device to perform a method according to the first aspect. The computer storage medium may be a non-transitory computer storage medium.

[0122] In a fourth aspect, the subject matter described herein provides a non-transitory computer storage medium having machine-executable instructions stored thereon, which when executed by a device cause the device to perform the method according to the first aspect.

[0123] The functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0124] The program code for performing the method for the subject matter described herein can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code causes the functions / operations specified in the flow chart and / or block diagram to be implemented when executed by the processor or controller. The program code can be executed entirely or partially on the machine, as a stand-alone software package partially on the machine, partially on a remote machine, or entirely on a remote machine or server.

[0125] In the context of the present disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media will include an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] In addition, although the operations are described in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in sequence, or to perform all the operations shown to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the subject matter described herein, but rather as descriptions of features that may be specific to a particular implementation. Certain features described in the context of separate implementations may also be implemented in combination in a single implementation. On the contrary, the various features described in a single implementation may also be implemented separately in multiple implementations or in any suitable sub-combination.

[0127] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A computer-implemented method for text line detection in an image, comprising: determining a first text region and a second text region in the image, the first text region including at least a first portion of a first text element and the second text region including at least a second portion of a second text element; extracting a first feature representation from the first text region and extracting a second feature representation from the second text region, the first feature representation and the second feature representation comprising semantic feature representations of text content included in the image; determining, based at least in part on the first feature representation and the second feature representation, a link relationship between the first text region and the second text region, the link relationship indicating whether the first portion of the first text element and the second portion of the second text element are located in a same text line; Determine the link relationship between the first feature representation and the second feature representation using a machine learning model; Determining a first text line region that at least defines the first text region and the second text region in the image based on the link relationship indicating that at least a portion of the first text element in the first text region and at least a portion of the second text element in the second text region are located in the same text line; as well as The first text line area is provided.

2. The method of claim 1, wherein extracting a first feature representation from the first text region and extracting a second feature representation from the second text region comprises: identifying at least the first text element from the first text region, and identifying at least the second text element from the second text region; as well as A first semantic feature representation is generated based on the identified first text element, and a second semantic feature representation is generated based on the identified second text element.

3. The method of claim 1, wherein determining the first text region and the second text region comprises: identifying a plurality of text regions from the image; as well as The first text region and the second text region are selected from the plurality of text regions according to at least one of the following: determining that a difference between a first size of the first text region and a second size of the second text region is below a size threshold, or It is determined that a first distance between a first center of the first text region and a second center of the second text region is below a distance threshold.

4. The method according to claim 1, wherein determining the link relationship between the first text region and the second text region comprises: extracting a third feature representation indicating at least one of: a spatial relationship between the first text region and the second text region, or a semantic relationship between the first text element and the second text element; and The link relationship is further determined based on the third feature representation.

5. The method according to claim 4, wherein extracting the third feature representation comprises: determining a third text region that at least bounds the first text region and the second text region; as well as The third feature representation is extracted from the third text region.

6. The method according to claim 4, wherein extracting the third feature representation comprises: Obtaining a first coordinate set associated with the first text area and a second coordinate set associated with the second text area; as well as The third feature representation is determined based on the first set of coordinates and the second set of coordinates.

7. The method according to claim 1, wherein providing the first text line area comprises: determining a second text line region in the image, wherein an overlapping area between the first text line region and the second text line region exceeds an area threshold; extracting a fourth feature representation from the first text line region, and extracting a fifth feature representation from the second text line region; the combination of the fourth feature representation and the fifth feature representation is configured to be provided to a relationship prediction model for determining which candidate text line region to provide, wherein the relationship prediction model is trained by selecting text line region pairs, wherein the overlapping area between the text line region pairs exceeds an area threshold, and is trained on true values ​​based on user selections; determining whether the first text line region is to be provided and not the second text line region based at least in part on the fourth feature representation and the fifth feature representation; as well as According to determining that the first text line region is to be provided, the first text line region is provided in the image, but the second text line region is not provided.

8. The method according to claim 1, wherein providing the first text line area comprises: The first text line area is highlighted in the image.

9. The method of claim 1, wherein the first text element comprises at least one of a character or a symbol.

10. An electronic device comprising: Processing unit; as well as a memory coupled to the processing unit and having instructions stored thereon, the instructions, when executed by the processing unit, causing the electronic device to perform actions, the actions comprising: determining a first text region and a second text region in the image, the first text region including at least a first portion of a first text element and the second text region including at least a second portion of a second text element; extracting a first feature representation from the first text region and extracting a second feature representation from the second text region, the first feature representation and the second feature representation comprising semantic feature representations of text content included in the image; determining a link relationship between the first text region and the second text region based at least in part on the first feature representation and the second feature representation, the link relationship indicating whether the first portion and the second portion of the first text element and the second text element are located in a same text line; Determine the link relationship between the first feature representation and the second feature representation using a machine learning model; Determining a first text line region that at least defines the first text region and the second text region in the image based on the link relationship indicating that at least a portion of the first text element in the first text region and at least a portion of the second text element in the second text region are located in the same text line; and The first text line area is provided.

11. The electronic device of claim 10, wherein extracting a first feature representation from the first text region and extracting a second feature representation from the second text region comprises: identifying at least the first text element from the first text region, and identifying at least the second text element from the second text region; as well as A first semantic feature representation is generated based on the identified first text element, and a second semantic feature representation is generated based on the identified second text element.

12. The electronic device of claim 10, wherein determining the first text area and the second text area comprises: identifying a plurality of text regions from the image; as well as The first text region and the second text region are selected from the plurality of text regions according to at least one of the following: determining that a difference between a first size of the first text region and a second size of the second text region is below a size threshold, or It is determined that a first distance between a first center of the first text region and a second center of the second text region is below a distance threshold.

13. The electronic device according to claim 10, wherein determining the link relationship between the first text area and the second text area comprises: extracting a third feature representation indicating at least one of: a spatial relationship between the first text region and the second text region, or a semantic relationship between the first text element and the second text element; and The link relationship is further determined based on the third feature representation.

14. A computer program product tangibly stored on a computer storage medium and comprising machine executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 9.

Images