Paragraph detection method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a paragraph detection method and device, electronic equipment and storage medium, comprising: obtaining text line instances in a target text picture, wherein the target text picture is a picture including text and needing paragraph detection, and the text line instances are text belonging to the same line or text belonging to the same column; clustering all text line instances according to instance features corresponding to each text line instance to determine at least one target paragraph in the target text picture, wherein the instance features are used to indicate the visual representation and / or position of the corresponding text, and the target paragraph includes at least one text line instance. Through the present application, the technical problems of easy paragraph omission and additional calculation in the related art can be overcome.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and more specifically, the embodiments of the present invention relate to paragraph detection methods and apparatus, electronic devices, and storage media. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention as set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] The paragraph detection task in general OCR tasks specifically involves locating text regions with independent semantic units (paragraphs) on an input image.

[0004] In related technologies, paragraph detection in document images mainly involves detecting text paragraphs within the input document image. A common approach is top-down (detecting paragraphs first, then individual text lines within those paragraphs), using general object detection methods such as YOLO and Mask-RCNN. However, these methods are prone to missing paragraphs. Furthermore, both top-down paragraph detection and subsequent text detection tasks involve a preceding convolutional neural network feature extraction stage, leading to task overlap and additional computational overhead.

[0005] It is evident that the paragraph detection methods in the relevant technologies have technical problems such as the tendency to miss paragraphs and the introduction of additional computational load. Summary of the Invention

[0006] However, due to the need to accurately detect each paragraph and reduce computational load, existing methods for paragraph detection in documents cannot meet these requirements because they suffer from missed paragraphs and incur additional computational load.

[0007] Therefore, in existing technologies, paragraph detection of documents is prone to missing detections, and because there is overlap between top-down paragraph detection and subsequent text detection tasks, it brings additional computational load, which is a very troublesome process.

[0008] Therefore, there is a great need for an improved paragraph detection method to accurately detect all paragraphs in a document and reduce computational load.

[0009] In this context, embodiments of the present invention aim to provide a paragraph detection method, including:

[0010] Obtain text line instances from a target text image, wherein the target text image is an image containing text and requiring paragraph detection, and the text line instances are text belonging to the same line or text belonging to the same column;

[0011] Based on the instance features corresponding to each text line instance, all text line instances are clustered to determine at least one target paragraph in the target text image, wherein the instance features are used to indicate the visual representation and / or location of the corresponding text, and the target paragraph includes at least one of the text line instances.

[0012] In another embodiment of the present invention, as described above, the step of clustering all text line instances according to the instance features corresponding to each text line instance to determine at least one target paragraph in the target text image includes:

[0013] Visual features are extracted from the target text image to obtain a visual feature map;

[0014] The target visual features of each text line instance are determined from the visual feature map;

[0015] Determine the positional features corresponding to the position of each text line instance in the target text image;

[0016] Based on the target visual features and the positional features, all text line instances are clustered to determine at least one target paragraph.

[0017] In another embodiment of the present invention, as described above, determining the target visual features of each text line instance from the visual feature map includes:

[0018] For each of the text line instances, an initial visual feature representation of each text line instance is obtained from the visual feature map using a region of interest feature extraction algorithm;

[0019] By performing mean pooling on the initial visual feature representation of each text line instance, a general visual feature of each text line instance is obtained, wherein the general visual feature of each text line instance is used to indicate the general visual information of each text line instance;

[0020] Based on the initial visual feature representation of each text line instance, characteristic visual features of each text line instance are determined, wherein the characteristic visual features include features used to indicate that they belong to the same paragraph;

[0021] The general visual features and the specific visual features are spliced ​​together according to channels to obtain the target visual features.

[0022] In another embodiment of the present invention, as described in the foregoing method, determining the characteristic visual features of each text line instance according to the initial visual feature representation of each text line instance includes:

[0023] Each text line instance is represented by an initial embedding vector;

[0024] Adaptive modeling is performed on the initial embedding vector and the initial visual feature representation to obtain the characteristic visual features of the text line instance.

[0025] In another embodiment of the present invention, as described above, the adaptive modeling of the initial embedding vector and the initial visual feature representation to obtain the characteristic visual features of the text line instance includes:

[0026] The initial visual feature representation of each text line instance is converted into a feature map vector of a target dimension, wherein the target dimension is the number of dimensions of the initial embedding vector;

[0027] The feature map vector is concatenated with the initial embedding vector to obtain the first concatenated vector;

[0028] The first concatenated vector is used to extract features based on a self-attention mechanism to obtain the characteristic visual features.

[0029] In another embodiment of the present invention, as described in the foregoing method, the step of clustering all text line instances based on the target visual features and the positional features to determine at least one target paragraph includes:

[0030] Based on the target visual features and the positional features corresponding to each text line instance, the final fusion feature of each text line instance is determined;

[0031] Input all the final fused features into the relationship determination model to obtain the determination information between every two text line instances, wherein the determination information is used to indicate whether the corresponding two text line instances belong to the same paragraph;

[0032] By clustering the text line instances belonging to the same paragraph according to all the determination information, at least one target paragraph is obtained.

[0033] In another embodiment of the present invention, as described above, the step of inputting all the final fused features into the relationship determination model to obtain the determination information between every two text line instances includes:

[0034] All the final fused features are input into the relation inference network in the relation determination model, and the relation inference network outputs the representation vector of each final fused feature, wherein the representation vector is the vector obtained by fusing the corresponding final fused feature with other final fused features through self-attention operation;

[0035] The result of concatenating every two representation vectors is input into the instance relation determiner in the relation determination model to obtain a classification result indicating whether every two representation vectors belong to the same paragraph;

[0036] Based on the text line instance corresponding to each of the representation vectors and the classification result, the determination information between every two text line instances is obtained.

[0037] In another embodiment of the present invention, as described above, determining the final fusion feature of each text line instance based on the target visual features and the positional features corresponding to each text line instance includes:

[0038] The target visual features and the positional features are concatenated according to channels to obtain a second concatenated vector;

[0039] The adaptive weights corresponding to the second concatenated vector are determined by a preset fully connected layer, and the second concatenated vector is weighted by the adaptive weights to obtain the final fusion feature of the text line instance.

[0040] In a second aspect of the present invention, a paragraph detection device is provided, comprising:

[0041] The acquisition module is used to acquire text line instances in a target text image, wherein the target text image is an image containing text and requiring paragraph detection, and the text line instances are text belonging to the same line or text belonging to the same column;

[0042] A clustering module is used to cluster all text line instances according to the instance features corresponding to each text line instance, and determine at least one target paragraph in the target text image, wherein the instance features are used to indicate the visual representation and / or position of the corresponding text, and the target paragraph includes at least one of the text line instances.

[0043] In another embodiment of the present invention, the clustering module, as described above, includes:

[0044] The visual feature extraction submodule is used to extract visual features from the target text image to obtain a visual feature map.

[0045] The visual feature determination submodule is used to determine the target visual features of each text line instance from the visual feature map;

[0046] The location feature determination submodule is used to determine the location features corresponding to the position of each text line instance in the target text image;

[0047] The clustering submodule is used to cluster all text line instances based on the target visual features and the positional features to determine at least one target paragraph.

[0048] In another embodiment of the present invention, as described in the foregoing apparatus, the visual feature determination submodule includes:

[0049] An initial visual feature representation unit is used to obtain the initial visual feature representation of each text line instance from the visual feature map using a region of interest feature extraction algorithm for each text line instance.

[0050] A general visual feature unit is used to obtain a general visual feature of each text line instance by performing mean pooling on the initial visual feature representation of each text line instance, wherein the general visual feature of each text line instance is used to indicate the general visual information of each text line instance;

[0051] A characteristic visual feature unit is used to determine the characteristic visual features of each text line instance according to the initial visual feature representation of each text line instance, wherein the characteristic visual features include features for indicating that they belong to the same paragraph;

[0052] The stitching unit is used to stitch the general visual features and the characteristic visual features according to channels to obtain the target visual features.

[0053] In another embodiment of the present invention, as described in the foregoing apparatus, the characteristic visual feature unit includes:

[0054] A characterization subunit is used to characterize each text line instance through an initial embedding vector;

[0055] An adaptive modeling subunit is used to adaptively model the initial embedding vector and the initial visual feature representation to obtain the characteristic visual features of the text line instance.

[0056] In another embodiment of the invention, as described above, the adaptive modeling subunit is configured to:

[0057] The initial visual feature representation of each text line instance is converted into a feature map vector of a target dimension, wherein the target dimension is the number of dimensions of the initial embedding vector;

[0058] The feature map vector is concatenated with the initial embedding vector to obtain the first concatenated vector;

[0059] The first concatenated vector is used to extract features based on a self-attention mechanism to obtain the characteristic visual features.

[0060] In another embodiment of the present invention, the clustering submodule, as described above, includes:

[0061] The final fusion feature determination unit is used to determine the final fusion feature of each text line instance based on the target visual features and the positional features corresponding to each text line instance.

[0062] The determination unit is used to input all the final fused features into the relationship determination model to obtain determination information between every two text line instances, wherein the determination information is used to indicate whether the corresponding two text line instances belong to the same paragraph;

[0063] A clustering unit is used to cluster the text line instances belonging to the same paragraph according to all the determination information to obtain at least one target paragraph.

[0064] In another embodiment of the present invention, as described in the foregoing apparatus, the final fusion feature determination unit includes:

[0065] The splicing subunit is used to splice the target visual features and the positional features according to channels to obtain a second spliced ​​vector;

[0066] The final fusion feature determination subunit is used to determine the adaptive weights corresponding to the second concatenated vector through a preset fully connected layer, and to weight the second concatenated vector with the adaptive weights to obtain the final fusion features of the text line instance.

[0067] In another embodiment of the present invention, as described in the foregoing apparatus, the determining unit is configured to:

[0068] All the final fused features are input into the relation inference network in the relation determination model, and the relation inference network outputs the representation vector of each final fused feature, wherein the representation vector is the vector obtained by fusing the corresponding final fused feature with other final fused features through self-attention operation;

[0069] The result of concatenating every two representation vectors is input into the instance relation determiner in the relation determination model to obtain a classification result indicating whether every two representation vectors belong to the same paragraph;

[0070] Based on the text line instance corresponding to each of the representation vectors and the classification result, the determination information between every two text line instances is obtained.

[0071] In a third aspect of the present invention, an electronic device is provided, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus, characterized in that...

[0072] The memory is used to store computer programs;

[0073] The processor is configured to perform the steps of the method described in any of the foregoing embodiments by running the computer program stored in the memory.

[0074] In a fourth aspect of the present invention, a computer-readable storage medium is provided, characterized in that the storage medium stores a computer program, wherein the computer program is configured to execute the steps of the method described in any of the preceding embodiments when it is run.

[0075] In this embodiment of the invention, all text line instances are clustered according to the instance features corresponding to each text line instance to determine at least one target paragraph in the target text image. Therefore, the target paragraph in the target text image can be determined directly based on the already determined text line instances, thus eliminating the need for pre-convolutional neural network feature extraction during paragraph detection. This effectively reduces the number of text detections and the amount of computation. Furthermore, since the detection rate of text line instances is higher than that of direct paragraph detection, the detection rate of paragraph detection can also be effectively improved. This overcomes the technical problems of paragraph omission and additional computation in related technologies. Attached Figure Description

[0076] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0077] Figure 1 A schematic flowchart of a paragraph detection method according to an optional embodiment of the present invention is shown.

[0078] Figure 2 A schematic flowchart of a paragraph detection method according to another optional embodiment of the present invention is shown;

[0079] Figure 3A schematic diagram of the structure of an optional paragraph detection model according to an embodiment of the present invention is shown.

[0080] Figure 4 A paragraph detection device according to an exemplary embodiment of the present invention is illustrated schematically;

[0081] Figure 5 An electronic device according to an exemplary embodiment of the present invention is illustrated schematically.

[0082] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0083] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art. Invention Overview

[0085] The inventors have discovered that paragraph detection in document images primarily involves detecting text paragraphs within the input document image. A common approach is a top-down method (detecting paragraphs first, then individual text lines within those paragraphs), utilizing general object detection methods such as YOLO and Mask-RCNN. However, these methods suffer from several drawbacks. First, they are prone to missing paragraphs. Second, both top-down paragraph detection and subsequent text detection tasks involve a preceding convolutional neural network feature extraction stage, leading to task overlap and additional computational overhead.

[0086] This invention provides a paragraph detection method, comprising: acquiring text line instances in a target text image, wherein the target text image is an image containing text and requiring paragraph detection, and the text line instances are text belonging to the same line or the same column; clustering all text line instances according to instance features corresponding to each text line instance to determine at least one target paragraph in the target text image, wherein the instance features are used to indicate the visual representation and / or position of the corresponding text, and the target paragraph includes at least one text line instance. In this way, clustering can be performed directly based on the already determined text line instances to determine the target paragraph in the target text image, thus eliminating the need for pre-convolutional neural network feature extraction during paragraph detection, effectively reducing the number of text detection operations and computational load; furthermore, since the detection rate of text line instances is higher than that of direct paragraph detection, the detection rate of paragraph detection can also be effectively improved; thus overcoming the technical problems of easy paragraph omission and additional computational load in related technologies.

[0087] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0088] Application Scenarios Overview

[0089] In this embodiment, the paragraph detection method described above can be applied to a hardware environment consisting of a terminal and a server. The server connects to the terminal via a network and can provide services to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services to the server.

[0090] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth.

[0091] First, the embodiments of the present invention can be implemented in various application scenarios. These scenarios include terminals and / or servers. Terminals can be mobile terminals, such as mobile electronic devices like mobile phones, tablets, laptops, and personal digital assistants, or desktop computers. Servers can be independent servers or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Communication between the terminal and the server can be established, for example, via mobile internet.

[0092] The paragraph detection process can be performed on the server side or the terminal side. More specifically, for example, using software installed on the terminal for paragraph detection, the user operates the software to obtain view information for each target view in the target response chain, thereby achieving the goal of obtaining the view information of its ancestor views simultaneously by locating a single view.

[0093] Exemplary methods

[0094] The following section, in conjunction with the above application scenarios, provides reference... Figure 1 This section describes a method for view inspection according to an exemplary embodiment of the present invention. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the invention, and the embodiments of the invention are not limited in any way. Rather, the embodiments of the invention can be applied to any applicable scenario.

[0095] This invention is applicable to scenarios where paragraphs of various sentences are detected in text images, but is not limited thereto.

[0096] For example, see Figure 1 The diagram shown is a flowchart of a paragraph detection method according to an embodiment of the present invention.

[0097] like Figure 1 As shown, the method may include the following steps:

[0098] Step S101: Obtain text line instances from the target text image. The target text image is an image containing text that needs to be detected as paragraphs. The text line instances are text belonging to the same line or text belonging to the same column.

[0099] The method described in this embodiment can be applied to scenarios where the translated text is displayed according to the paragraphs of the original text during the text translation process; or to scenarios where it is necessary to determine each paragraph of the identified text during the text extraction process.

[0100] In this embodiment, the target text image can be obtained by capturing images of paper materials, or by taking a screenshot or photograph of an image containing text displayed on an electronic device. In addition, the target text image can also be an image obtained by other means, as long as it contains text and requires paragraph detection. These are not listed here.

[0101] A text line instance can be obtained by performing text line detection on a target text image in a text detection task (i.e., a task of translating the original text). A text line instance can indicate a line of text or a column of text.

[0102] Step S102: Cluster all text line instances according to the instance features corresponding to each text line instance to determine at least one target paragraph in the target text image, wherein the instance features are used to indicate the visual representation and / or location of the corresponding text, and the target paragraph includes at least one text line instance.

[0103] After obtaining each text line instance, the instance characteristics corresponding to each text line instance can be determined.

[0104] Instance features can be features that represent the visual representation and / or position of the corresponding text line instance, and may include, but are not limited to, information such as font, font size, color, and position in the target text image.

[0105] After determining the instance features corresponding to each text line instance, text line instances with the same or similar instance features can be clustered into a target paragraph.

[0106] Furthermore, if, based on the instance characteristics, no other text line instance that needs to be clustered into the same target paragraph as text line instance I is identified, a corresponding target paragraph can be obtained directly based on text line instance I.

[0107] The method in this embodiment clusters all text line instances according to the instance features corresponding to each text line instance to determine at least one target paragraph in the target text image. Therefore, the target paragraph in the target text image can be determined directly based on the already determined text line instances, thus eliminating the need for pre-convolutional neural network feature extraction during paragraph detection. This effectively reduces the number of text detections and the amount of computation. Furthermore, since the detection rate of text line instances is higher than that of direct paragraph detection, the detection rate of paragraph detection can also be effectively improved. This overcomes the technical problems of paragraph omission and additional computation in related technologies.

[0108] like Figure 2 As shown, in another embodiment of the present invention, as described in the foregoing method, step S102 clusters all text line instances according to the instance features corresponding to each text line instance to determine at least one target paragraph in the target text image, including the following steps:

[0109] Step S201: Extract visual features from the target text image to obtain a visual feature map.

[0110] In this embodiment, the visual feature map of the target text image can be obtained by inputting the target text image into a lightweight fully convolutional neural network (i.e., Fully Convolutional Networks for Semantic Segmentation) for visual feature extraction.

[0111] Step S202: Determine the target visual features of each text line instance from the visual feature map.

[0112] After the visual feature map is determined, the target visual features can be determined on the visual feature map based on the various text line instances detected on the target text image, and for each text line instance.

[0113] The target visual features can be visual features such as color, font, font size, such as the serial number of a separate paragraph, list break information, and punctuation at the end of the text line, which are used to indicate the corresponding text line instance.

[0114] Step S203: Determine the positional features corresponding to the position of each text line instance in the target text image.

[0115] Furthermore, in order to more accurately cluster text line instances belonging to the same paragraph, this embodiment also determines the positional features corresponding to the position of each text line instance in the target text image.

[0116] The location features can be determined using either of the following two methods:

[0117] The bounding boxes of the text line detection results encode the position information of the text line instances through linear mapping. Specifically, two encoding methods can be used: one is to map the vector containing the width, height, center point coordinates, and rotation angle information of the text line instance through a fully connected layer to obtain the position features of the text line instance; the other is to embed the width, height, center coordinates, and rotation angle of the text line instance through an embedding layer, then concatenate them according to the channel dimension, and finally reduce the channel dimension to obtain the position features of the text line instance. The two methods have the same effect.

[0118] Optionally, this embodiment uses the first fully connected layer mapping method described above to determine the location features.

[0119] In other words, instance features include at least target visual features and location features.

[0120] Step S204: Based on the target's visual features and location features, cluster all text line instances to determine at least one target paragraph.

[0121] After determining the target visual features and positional features corresponding to each text line instance, all acquired text line instances can be clustered according to the target visual features and positional features of each text line instance to identify one or more text line instances belonging to the same paragraph, thereby identifying at least one target paragraph.

[0122] The method in this embodiment can cluster each text line instance based on visual and positional features, thereby effectively improving the accuracy of obtaining the target paragraph based on text line instance clustering.

[0123] In another embodiment of the present invention, as described above, step S202, which determines the target visual features of each text line instance from the visual feature map, includes the following steps:

[0124] Step S301: For each text line instance, obtain the initial visual feature representation of each text line instance from the visual feature map using the region of interest feature extraction algorithm.

[0125] In other words, after determining each text line instance, the initial visual feature representation of each text line instance can be obtained on the aforementioned visual feature map by using the region of interest feature extraction algorithm, that is, the visual feature representation of the image region corresponding to each text line instance.

[0126] Step S302: By performing mean pooling on the initial visual feature representation of each text line instance, a general visual feature of each text line instance is obtained, wherein the general visual feature of each text line instance is used to indicate the general visual information of each text line instance.

[0127] After determining the initial visual feature representation for each text line instance, mean pooling (averaging all values ​​in the local receptive domain) can be applied to the initial visual feature representation to obtain the general visual features corresponding to each text line instance. The general visual features include information such as the color, font, and font size of the text in the text line instance. Generally, text instances belonging to the same paragraph usually have the same general visual features. Therefore, the general visual features play a crucial role in determining whether different text line instances belong to the same paragraph.

[0128] Step S303: Based on the initial visual feature representation of each text line instance, determine the characteristic visual features of each text line instance, wherein the characteristic visual features include features used to indicate that they belong to the same paragraph.

[0129] In this embodiment, feature extraction is further performed based on the initial visual feature representation of each text line instance to obtain the characteristic visual features corresponding to each text line instance.

[0130] Visual features can be used to indicate whether different text line instances belong to the same paragraph. Furthermore, visual features can characterize the paragraph relationship of the current text line instance, such as the serial number before the text line instance, the punctuation at the end, etc.

[0131] Step S304: The general visual features and the characteristic visual features are spliced ​​together according to the channels to obtain the target visual features.

[0132] Optionally, the target visual feature can be obtained by concatenating the vectors corresponding to the general visual features and the vectors corresponding to the specific visual features.

[0133] In another embodiment of the present invention, as described above, step S303, based on the initial visual feature representation of each text line instance, determines the characteristic visual features of each text line instance, including the following steps:

[0134] Step S401: Characterize each text line instance using an initial embedding vector.

[0135] Optionally, an embedding vector (i.e., an initial embedding vector) can be initialized for each text line instance to represent the visual features of each text line instance. Furthermore, the same embedding vector can be used to represent the visual features of different text line instances.

[0136] Step S402: Adaptively model the initial embedding vector and the initial visual feature representation to obtain the characteristic visual features of the text line instance.

[0137] In another embodiment of the present invention, this step adaptively models the initial embedding vector and the initial visual feature representation to obtain the characteristic visual features of the text line instance, including the following steps:

[0138] Step S501: Convert the initial visual feature representation of each text line instance into a feature map vector of the target dimension number, where the target dimension number is the number of dimensions of the initial embedding vector.

[0139] In general, the initial embedding vector is a one-dimensional vector (i.e., the target dimension is one-dimensional). Therefore, the initial visual feature representation can also be converted into a one-dimensional feature map vector.

[0140] Step S502: Concatenate the feature map vector with the initial embedding vector to obtain the first concatenated vector.

[0141] After obtaining the feature map vector, the feature map vector can be concatenated with the initial embedding vector. For example, the feature map vector can be concatenated to the initial embedding vector to obtain the first concatenated vector.

[0142] Step S503: Based on the self-attention mechanism, feature extraction is performed on the first concatenated vector to obtain characteristic visual features.

[0143] After obtaining the first concatenated vector, it can be fed into the self-attention structure for feature extraction. Thus, the initial embedding vector (i.e., the embedding vector) will be aggregated to obtain visual information of the characteristics that are important for paragraph determination in the text line instances, such as the sequence number of a separate paragraph or list division information.

[0144] The method in this embodiment can extract characteristic visual features that represent belonging to the same paragraph based on the initial visual feature representation, thereby improving the accuracy of obtaining the target paragraph based on text line instance clustering in the later stage.

[0145] In another embodiment of the present invention, as described above, step S204 clusters all text line instances based on target visual features and positional features to determine at least one target paragraph, including the following steps:

[0146] Step S601: Based on the target visual features and positional features corresponding to each text line instance, determine the final fusion features of each text line instance.

[0147] After determining the target visual features and positional features corresponding to each text line instance, the target visual features and positional features of each text line instance can be spliced ​​and fused to obtain the final fused features that include both target visual features and positional features.

[0148] In another embodiment of the present invention, as described above, step S601 determines the final fusion feature of each text line instance based on the target visual features and positional features corresponding to each text line instance, including the following steps:

[0149] Step S701: The target visual features and positional features are concatenated according to channels to obtain the second concatenated vector;

[0150] For example, a one-dimensional first vector and a one-dimensional second vector are obtained based on the target's visual features and positional features, respectively. Then, the first vector and the second vector are concatenated to obtain the second concatenated vector.

[0151] Step S702: Determine the adaptive weights corresponding to the second concatenated vector through a preset fully connected layer, and weight the second concatenated vector using the adaptive weights to obtain the final fusion features of the text line instances.

[0152] After obtaining the second concatenated vector, in order to determine the weights corresponding to the target visual features and positional features respectively (i.e., the importance of the target visual features and positional features in paragraph judgment), the second concatenated vector can be input into a preset fully connected layer. Then, the preset fully connected layer weights the first and second vectors in the second concatenated vector to obtain the weights corresponding to the first and second vectors respectively, thereby obtaining the final fused features of the text line instance.

[0153] Step S602: Input all the final fused features into the relationship determination model to obtain the determination information between every two text line instances. The determination information is used to indicate whether the corresponding two text line instances belong to the same paragraph.

[0154] Optionally, the relationship determination model can determine the similarity between any two final fused features, and thus obtain the determination information corresponding to the two text line instances corresponding to any two final fused features.

[0155] Optionally, the determination information can be used to characterize whether any two text line instances belong to the same paragraph, and further, it can also include the confidence level that they belong to the same paragraph.

[0156] In another embodiment of the present invention, as described above, all final fused features are input into the relationship determination model to obtain determination information between every two text line instances, including the following steps:

[0157] Step S801: Input all final fused features into the relation inference network in the relation determination model. The relation inference network outputs the representation vector of each final fused feature. The representation vector is the vector obtained by fusing other final fused features through self-attention operation on the corresponding final fused feature.

[0158] The common approach to determining the relationship between two vectors is to calculate their cosine similarity; a higher cosine similarity indicates a stronger correlation between the two vectors. In this invention, since text line instance vectors contain not only visual information but also positional information, text instances belonging to the same paragraph exhibit high visual cosine similarity. However, a higher cosine similarity in positional information indicates closer proximity between the two locations, and proximity alone cannot definitively determine whether two text line instances belong to the same paragraph.

[0159] Therefore, this embodiment proposes using a learnable relation determination model to determine the relationships between text instances. Furthermore, the relation determination model includes a relation reasoning network and an instance relation determiner.

[0160] In this embodiment, after the final fused features of the text line instances are input into the relationship determination model, the final fused features of all text line instances are first sent to the relationship inference network of the relationship determination model to perform relationship inference on whether the text line instances belong to the same paragraph.

[0161] Optionally, the relational reasoning network is composed of a GAT graph attention network or a self-attention structure. Both GAT graph attention networks and self-attention structures can play a role in relation extraction. This invention employs a self-attention structure to maximize the receptive field between text line instances. All text line instances are modeled in terms of their relationships with each other within the relational reasoning network. Each text line instance calculates attention weights with other text line instances and performs feature weighting on the final fused features of other text line instances. Text line instances belonging to the same paragraph have a higher correlation. After modeling the relationships between text line instances through the relational reasoning network, a representation vector corresponding to each text line instance is obtained.

[0162] Using relational reasoning networks can improve the generalization of the structure and increase the accuracy of determining the relationships between text instances.

[0163] Step S802: The result of concatenating every two representation vectors is input into the instance relation determiner in the relation determination model to obtain a classification result indicating whether every two representation vectors belong to the same paragraph.

[0164] In this embodiment, after obtaining the representation vector corresponding to each text line instance, the final step is to use an instance relationship determiner to determine whether the text line instances belong to the same paragraph.

[0165] Optionally, the instance relation determiner can be a classifier consisting of three fully connected layers. After each pair of representation vectors is input into the instance determiner, the instance determiner can classify the relationship between each pair of representation vectors and obtain a classification result indicating whether each pair of representation vectors belongs to the same paragraph.

[0166] Step S803: Based on the text line instance corresponding to each representation vector and the classification result, obtain the judgment information between every two text line instances.

[0167] Since the representation vector uniquely corresponds to a text line instance, the classification result can indicate whether two text line instances corresponding to two representation vectors belong to the same paragraph.

[0168] Therefore, after obtaining all classification results, the judgment information between each pair of text line instances can be determined based on the text line instance corresponding to each representation vector.

[0169] For example, if the classification result corresponding to two representation vectors is 0, we can obtain the determination information that the two text line instances corresponding to the two representation vectors do not belong to the same paragraph; if the classification result corresponding to two representation vectors is 1, we can obtain the determination information that the two text line instances corresponding to the two representation vectors belong to the same paragraph.

[0170] Step S603: Cluster the text line instances belonging to the same paragraph according to all the determination information to obtain at least one target paragraph.

[0171] After obtaining all the decision information, the text line instances belonging to the same paragraph can be clustered using the disjoint-set data structure algorithm to obtain the complete text paragraph (i.e., the target paragraph).

[0172] like Figure 3 As shown, an optional paragraph detection model for implementing the methods of the aforementioned embodiments is provided. The paragraph detection model structure mainly consists of five modules: a lightweight fully convolutional backbone network, a visual information extraction module, a positional information encoding module, a relation reasoning network, and an instance relation determiner. Figure 3 The input target image, text line detection results, and specific text content in the clustered text paragraphs cannot be used as specific limitations of this application, and even if the text content is changed, it will not affect the specific detection process of this paragraph detection model.

[0173] A fully convolutional backbone network is used to extract visual features from the whole image, resulting in a visual feature map.

[0174] The visual information extraction module is used to obtain the initial visual feature representation of each text line instance and to perform visual feature aggregation to obtain the target visual features.

[0175] The location information encoding module is used to perform operations such as embedding the location features of text line instances to obtain location features.

[0176] Relational reasoning networks are used to model the relationships between instances of text lines.

[0177] The instance relationship determiner determines whether any two text line instances belong to the same paragraph, and obtains the text line IDs (one text line ID corresponding to each text line instance) contained in each target paragraph based on the disjoint set processing algorithm.

[0178] The input to the entire paragraph detection model includes the input target text image and the text line detection results (i.e., each text line instance), and the output is the text line ID contained in each target paragraph.

[0179] When applying the paragraph detection model in this embodiment to a photo translation scenario, the scenario requires translating the source language content of an image into the target language (e.g., translating English into Chinese) and then re-rendering it back to the original image. Paragraph detection in the image is crucial for restoring the original layout structure; without paragraph detection, it's impossible to correlate the original and translated text areas in the image. The overall photo translation process already includes a text detection component. Using this bottom-up paragraph detection method, it's possible to directly determine whether detected text lines belong to the same paragraph, without needing to detect paragraph boxes and then use the IOU between the text line detection box and the text paragraph detection box to determine which text lines belong to the same paragraph. Specifically:

[0180] First, the bounding boxes of all text line instances are obtained through the text line detection model in the overall image translation process. The bounding boxes of the text line instance detection results and the original image are used as the two inputs to the paragraph detection model. The original image is processed by a lightweight fully convolutional backbone network to extract visual features, resulting in a visual feature map. Then, an initial visual feature representation of each text line instance is obtained through a region of interest feature extraction algorithm.

[0181] The initial visual feature representation uses the visual information extraction module to obtain the target visual features of the text line instance.

[0182] The coordinate boxes of the text line detection results are obtained by using the position information encoding module to obtain the position features of the text line instances. The information from the two modalities (i.e., target visual features and position features) are adaptively fused and the final fused features are fed into the relation inference network to model the relationship between each pair of text line instances and obtain the final representation vector of the text line instances. Finally, the text line instance representation vector is fed into the instance relation determiner to obtain the determination information of whether each pair of text line instances belongs to the same paragraph. The text line instances of the same paragraph are clustered by the disjoint set algorithm to obtain the text paragraph (i.e., the target paragraph).

[0183] The rendering module of the photo translation will render the corresponding translation in the corresponding area based on the text paragraphs obtained by paragraph detection, restoring the original layout structure.

[0184] Exemplary device

[0185] After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 4 An exemplary embodiment of the view inspection apparatus of the present invention will be described.

[0186] The implementation of the view inspection device can refer to the implementation of the method described above, and repeated details will not be repeated. The terms "unit" or "module" used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0187] This invention provides a view inspection device. Figure 4 The paragraph detection device according to an exemplary embodiment of the present invention is illustrated schematically, such as... Figure 4 As shown, the device includes:

[0188] Module 1 is used to obtain text line instances in the target text image. The target text image is an image that includes text and needs to be detected as paragraphs. The text line instances are text belonging to the same line or text belonging to the same column.

[0189] Clustering module 2 is used to cluster all text line instances according to the instance features corresponding to each text line instance, and determine at least one target paragraph in the target text image, wherein the instance features are used to indicate the visual representation and / or position of the corresponding text, and the target paragraph includes at least one text line instance.

[0190] In another embodiment of the invention, the clustering module, as described above, includes:

[0191] The visual feature extraction submodule is used to extract visual features from the target text image to obtain a visual feature map.

[0192] The visual feature determination submodule is used to determine the target visual features of each text line instance from the visual feature map;

[0193] The location feature determination submodule is used to determine the location features corresponding to the position of each text line instance in the target text image;

[0194] The clustering submodule is used to cluster all text line instances based on target visual features and positional features to identify at least one target paragraph.

[0195] In another embodiment of the invention, the visual feature determination submodule, as described above, includes:

[0196] An initial visual feature representation unit is used to obtain the initial visual feature representation of each text line instance from the visual feature map using a region of interest feature extraction algorithm, according to each text line instance.

[0197] A general visual feature unit is used to obtain the general visual features of each text line instance by mean pooling the initial visual feature representation of each text line instance, wherein the general visual features of each text line instance are used to indicate the general visual information of each text line instance.

[0198] A characteristic visual feature unit is used to determine the characteristic visual features of each text line instance according to the initial visual feature representation of each text line instance, wherein the characteristic visual features include features used to indicate that they belong to the same paragraph;

[0199] The splicing unit is used to splice general visual features and characteristic visual features according to channels to obtain the target visual features.

[0200] In another embodiment of the present invention, the aforementioned apparatus, the characteristic visual feature unit includes:

[0201] The representation subunit is used to represent the initial visual feature representation of each text line instance through the initial embedding vector;

[0202] The adaptive modeling subunit is used to adaptively model the initial embedding vector and the initial visual feature representation to obtain the characteristic visual features of the text line instance.

[0203] In another embodiment of the invention, the adaptive modeling subunit, as described above, is used for:

[0204] The initial visual feature representation of each text line instance is converted into a feature map vector of the target dimension, where the target dimension is the number of dimensions of the initial embedding vector.

[0205] The feature map vector is concatenated with the initial embedding vector to obtain the first concatenated vector;

[0206] Feature extraction is performed on the first concatenated vector based on the self-attention mechanism to obtain characteristic visual features.

[0207] In another embodiment of the invention, the clustering submodule of the apparatus described above includes:

[0208] The final fusion feature determination unit is used to determine the final fusion feature of each text line instance based on the target visual features and positional features corresponding to each text line instance.

[0209] The decision unit is used to input all the final fused features into the relationship decision model to obtain the decision information between every two text line instances. The decision information is used to indicate whether the corresponding two text line instances belong to the same paragraph.

[0210] Clustering units are used to cluster text line instances belonging to the same paragraph according to all decision information to obtain at least one target paragraph.

[0211] In another embodiment of the present invention, the apparatus as described above, the final fusion feature determination unit includes:

[0212] The splicing sub-unit is used to splice the target visual features and positional features according to channels to obtain the second spliced ​​vector;

[0213] The final fusion feature determination subunit is used to determine the adaptive weights corresponding to the second concatenated vector through a preset fully connected layer, and to weight the second concatenated vector with the adaptive weights to obtain the final fusion features of the text line instance.

[0214] In another embodiment of the present invention, as described above, the determining unit is configured to:

[0215] All final fused features are input into the relation inference network in the relation determination model. The relation inference network outputs the representation vector of each final fused feature. The representation vector is the vector obtained by fusing other final fused features through self-attention operation for the corresponding final fused feature.

[0216] The result of concatenating every two representation vectors is input into the instance relation determiner in the relation determination model to obtain a classification result indicating whether every two representation vectors belong to the same paragraph.

[0217] Based on the text line instance corresponding to each representation vector and the classification result, the judgment information between every two text line instances is obtained.

[0218] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0219] According to another embodiment of this application, an electronic device is also provided, comprising: Figure 5 As shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0220] Memory 1503 is used to store computer programs;

[0221] When the processor 1501 executes the program stored in the memory 1503, it implements the steps of the above method embodiment.

[0222] The bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0223] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0224] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0225] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0226] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method steps provided in any of the foregoing method embodiments.

[0227] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0228] It should be noted that although several modules, sub-modules, or units of the view inspection device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided and embodied by multiple devices.

[0229] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowchart may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0230] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that, when executed by a computer or other programmable data processing device, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0231] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0232] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0233] The use of the verbs "including" and "contains" and their inflections in the application documents does not preclude the existence of elements or steps other than those described in the application documents. The article "a" or "one" preceding an element does not preclude the existence of multiple such elements.

[0234] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A paragraph detection method, characterized in that, include: Obtain text line instances from a target text image, wherein the target text image is an image containing text and requiring paragraph detection, and the text line instances are text belonging to the same line or text belonging to the same column; Clustering all text line instances according to instance features corresponding to each text line instance to determine at least one target paragraph in the target text image includes: extracting visual features from the target text image to obtain a visual feature map; determining the target visual features of each text line instance from the visual feature map; determining the positional features corresponding to the position of each text line instance in the target text image; and clustering all text line instances according to the target visual features and the positional features to determine at least one target paragraph, wherein the instance features are used to indicate the visual representation and / or position of the corresponding text, and the target paragraph includes at least one text line instance; The step of clustering all text line instances based on the target visual features and the positional features to determine at least one target paragraph includes: Based on the target visual features and the positional features corresponding to each text line instance, the final fusion feature of each text line instance is determined; By inputting all the final fused features into the relationship determination model, the determination information between every two text line instances is obtained, wherein the determination information is used to indicate whether the corresponding two text line instances belong to the same paragraph; By clustering the text line instances belonging to the same paragraph according to all the determination information, at least one target paragraph is obtained.

2. The method according to claim 1, characterized in that, Determining the target visual features of each text line instance from the visual feature map includes: For each of the text line instances, an initial visual feature representation of each text line instance is obtained from the visual feature map using a region of interest feature extraction algorithm; By performing mean pooling on the initial visual feature representation of each text line instance, a general visual feature of each text line instance is obtained, wherein the general visual feature of each text line instance is used to indicate the general visual information of each text line instance. Based on the initial visual feature representation of each text line instance, characteristic visual features of each text line instance are determined, wherein the characteristic visual features include features for indicating that they belong to the same paragraph; The general visual features and the specific visual features are spliced ​​together according to channels to obtain the target visual features.

3. The method according to claim 2, characterized in that, The step of determining the characteristic visual features of each text line instance based on the initial visual feature representation of each text line instance includes: Each text line instance is represented by an initial embedding vector; Adaptive modeling is performed on the initial embedding vector and the initial visual feature representation to obtain the characteristic visual features of the text line instance.

4. The method according to claim 3, characterized in that, The adaptive modeling of the initial embedding vector and the initial visual feature representation to obtain the characteristic visual features of the text line instance includes: The initial visual feature representation of each text line instance is converted into a feature map vector of a target dimension, wherein the target dimension is the number of dimensions of the initial embedding vector; The feature map vector is concatenated with the initial embedding vector to obtain the first concatenated vector; The first concatenated vector is used to extract features based on a self-attention mechanism to obtain the characteristic visual features.

5. The method according to claim 1, characterized in that, The step of inputting all the final fused features into the relationship determination model to obtain the determination information between every two text line instances includes: All the final fused features are input into the relation inference network in the relation determination model, and the relation inference network outputs the representation vector of each final fused feature, wherein the representation vector is the vector obtained by fusing the corresponding final fused feature with other final fused features through self-attention operation; The result of concatenating every two representation vectors is input into the instance relation determiner in the relation determination model to obtain a classification result indicating whether every two representation vectors belong to the same paragraph; Based on the text line instance corresponding to each of the representation vectors and the classification result, the determination information between every two text line instances is obtained.

6. A paragraph detection device, characterized in that, include: The acquisition module is used to acquire text line instances in a target text image, wherein the target text image is an image containing text and requiring paragraph detection, and the text line instances are text belonging to the same line or text belonging to the same column; A clustering module is used to cluster all text line instances according to the instance features corresponding to each text line instance, and determine at least one target paragraph in the target text image. This includes: extracting visual features from the target text image to obtain a visual feature map; determining the target visual features of each text line instance from the visual feature map; determining the positional features corresponding to the position of each text line instance in the target text image; and clustering all text line instances according to the target visual features and the positional features to determine at least one target paragraph, wherein the instance features are used to indicate the visual representation and / or position of the corresponding text, and the target paragraph includes... At least one text line instance; the step of clustering all text line instances according to the target visual features and the positional features to determine at least one target paragraph includes: determining the final fusion feature of each text line instance based on the target visual features and the positional features corresponding to each text line instance; inputting all the final fusion features into a relationship determination model to obtain determination information between every two text line instances, wherein the determination information is used to indicate whether the corresponding two text line instances belong to the same paragraph; and clustering the text line instances belonging to the same paragraph according to all the determination information to obtain at least one target paragraph.

7. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, the communication interface, and the memory communicate with each other via the communication bus, characterized in that... The memory is used to store computer programs; The processor is configured to perform the method steps of any one of claims 1 to 5 by running the computer program stored in the memory.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the steps of the method described in any one of claims 1 to 5 when it is run.

Images