End-to-end document detection method and device, electronic equipment and storage medium

Abstract

The invention provides an end-to-end document detection method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like. According to the specific implementation scheme, feature extraction is conducted on an image corresponding to a target document, and an image feature sequence is obtained; paragraph detection is carried out on the target document, and the position and size parameters of a detection frame corresponding to each paragraph are determined; determining a text feature sequence corresponding to the target document according to the position and size parameters of each detection box; and performing fusion decoding on the image feature sequence and the text feature sequence to obtain a reading sequence of the target document.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and specifically to an end-to-end document detection method, device, electronic device, and storage medium. Background Art

[0002] Existing large model training relies on high-quality pre-training data. PDF (Portable Document Format) documents from professional scenarios such as papers, journals, and books are reliable data for large model training. By parsing PDF documents, large-scale document pre-training data can be produced to enhance the general knowledge and professional knowledge capabilities of large models. Therefore, improving the parsing ability of PDF documents is of great significance to improving the training effect of large models. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the purpose of the present disclosure is to propose an end-to-end document detection method, device, electronic device and storage medium, so as to achieve the sorting output of each paragraph box by using an encoder and a decoder based on the self-attention mechanism Transformer, thereby improving the speed and quality of determining the document reading order.

[0005] According to a first aspect of the present disclosure, an end-to-end document detection method is provided, comprising:

[0006] Extract features of the image corresponding to the target document to obtain an image feature sequence;

[0007] Perform paragraph detection on the target document to determine the position and size parameters of the detection box corresponding to each paragraph;

[0008] Determining a text feature sequence corresponding to the target document according to the position and size parameters of each detection frame;

[0009] The image feature sequence and the text feature sequence are fused and decoded to obtain the reading order of the target document.

[0010] According to a second aspect of the present disclosure, an end-to-end document detection device is provided, comprising:

[0011] The encoding module is used to extract features of the image corresponding to the target document to obtain an image feature sequence;

[0012] A first determination module is used to perform paragraph detection on the target document and determine the position and size parameters of the detection box corresponding to each paragraph;

[0013] A second determination module is used to determine the text feature sequence corresponding to the target document according to the position and size parameters of each detection frame;

[0014] The decoding module is used to fuse and decode the image feature sequence and the text feature sequence to obtain the reading order of the target document.

[0015] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the end-to-end document detection method as described in the first aspect.

[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the end-to-end document detection method as described in the first aspect.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the end-to-end document detection method as described in the first aspect.

[0021] The end-to-end document detection method, device, electronic device, and storage medium provided by the present disclosure have the following beneficial effects:

[0022] By extracting the graphic feature sequence corresponding to the document image and the text feature sequence determined by the position and size parameters of each paragraph detection box in the document, the image feature sequence and the text feature sequence are decoded to obtain the serial number corresponding to each paragraph in the document to determine the reading order of the document. End-to-end document detection is achieved without the need to identify the semantics of each paragraph. While ensuring the effect of document reading order detection, the speed of determining the reading order is improved.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, which are used to better understand the present solution and do not constitute a limitation of the present disclosure, wherein:

[0025] Figure 1 is a flowchart of an end-to-end document detection method proposed according to an embodiment of the present disclosure;

[0026] Figure 2 A schematic diagram of a process for detecting the reading order of each paragraph of a document by an end-to-end document detection model provided in an embodiment of the present disclosure;

[0027] Figure 3 is a flow chart of an end-to-end document detection method proposed according to another embodiment of the present disclosure;

[0028] Figure 4 is a flow chart of an end-to-end document detection method proposed according to another embodiment of the present disclosure;

[0029] Figure 5 is a flow chart of an end-to-end document detection method proposed according to another embodiment of the present disclosure;

[0030] Figure 6 This is an example diagram of the effect of a document reading order proposed by an embodiment of the present disclosure;

[0031] Figure 7 is a schematic diagram of the structure of an end-to-end document detection device proposed according to an embodiment of the present disclosure;

[0032] Figure 8 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] The embodiments of the present disclosure relate to the field of artificial intelligence technology, and in particular to technical fields such as computer vision, deep learning, and large models.

[0035] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0036] Computer vision refers to the use of cameras and computers to replace the human eye to identify, track and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection.

[0037] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images, and sounds.

[0038] Large models refer to "large parameter" models trained using large-scale data and powerful computing power. These models are usually highly versatile and generalizable. Large models can capture complex data patterns through their large number of parameters, deep network structure and extensive pre-training capabilities. They can not only understand and generate natural language, but also process complex visual and multimodal information and adapt to various and changing application scenarios.

[0039] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0040] It should be noted that the parsing of PDF documents depends on the capabilities of layout detection, text detection and recognition, and reading order detection. Since reading order detection is very challenging on complex PDF documents, the purpose of improving the parsing capabilities of PDF documents is to study a more efficient and reliable method for determining the reading order.

[0041] The following describes the end-to-end document detection method, device, electronic device and storage medium according to the embodiments of the present disclosure with reference to the accompanying drawings.

[0042] It should be noted that the executor of the end-to-end document detection method of this embodiment is an end-to-end document detection device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, and the electronic device may include but is not limited to a terminal, a server, etc.

[0043] Figure 1 It is a flowchart of an end-to-end document detection method proposed according to an embodiment of the present disclosure.

[0044] like Figure 1 As shown, the end-to-end document detection method includes:

[0045] S101: Extract features of the image corresponding to the target document to obtain an image feature sequence.

[0046] It should be noted that in the present disclosure, when the target document type is PDF, the PDF document itself is an image or a series of images, which can be directly used for feature extraction. Alternatively, when the target document is a document of other types (such as text word, table excel, etc.), the target document can be converted to PDF to obtain the image corresponding to the target document, which is then used for feature extraction.

[0047] In the disclosed embodiment, a visual encoder (such as a convolutional neural network (CNN) etc.) may be used to complete feature extraction by inputting an image corresponding to a target document into the visual encoder for encoding, thereby obtaining an image feature sequence output by the visual encoder. Specifically, the image may be firstly cut into image blocks of the same size, and then encoded through a convolutional network to obtain a sequence of multiple image blocks, thereby obtaining an image feature sequence.

[0048] It should be noted that in the present disclosure, for the visual encoder, in order to solve the coding efficiency problem of high-resolution images, a sliding window mechanism can also be introduced in the visual encoder, so that the visual encoder can learn information across windows. At the same time, through the downsampling layer, the visual encoder can process high-resolution images, saving computational complexity while paying attention to global and local information.

[0049] S102: Perform paragraph detection on the target document to determine the position and size parameters of the detection box corresponding to each paragraph.

[0050] The position of the detection frame may be the coordinates of any vertex or center point of the detection frame in the target document, for example, the 7th line of the second page, etc. The scale parameters of the detection frame may include the length, width, shape, etc. of the detection frame.

[0051] In the disclosed embodiment, each paragraph in the target document can be identified based on paragraph characteristics such as the first line indent. Every time a paragraph is detected, all the contents belonging to the paragraph can be selected by a rectangular border. The position and size parameters of the detection frame corresponding to each paragraph can be determined by obtaining coordinates of a specific point (such as the upper left corner) of the rectangular border, measuring the length and width, etc.

[0052] S103: Determine a text feature sequence corresponding to the target document according to the position and size parameters of each detection frame.

[0053] In the disclosed embodiment, after determining the detection box corresponding to each paragraph in the target document, the detection boxes corresponding to all paragraphs can be input into the decoder in a random order, and each detection box can be converted into an embedding vector according to the position and size parameters of the detection box, so as to obtain the text feature sequence corresponding to the target document.

[0054] S104: fusing and decoding the image feature sequence and the text feature sequence to obtain the reading order of the target document.

[0055] In the disclosed embodiment, a sequence decoder based on Transformer can be used to fuse and decode the image feature sequence and the text feature sequence, and the decoder outputs the serial number corresponding to each paragraph when reading the target document, so that the reading order of the target document can be obtained. Specifically, in the decoder, the text feature sequence can first be subjected to the interaction between different detection frames through the self-attention layer, and then interacted with the image feature sequence in the cross-attention layer. Finally, the decoder will predict the paragraph detection frame numbers that meet the correct reading order one by one in an autoregressive manner, thereby obtaining the serial number corresponding to each paragraph.

[0056] It should be noted that in the present disclosure, in order to enable the decoder part to be better initialized, the decoder of the pre-trained language model mBart (Multilingual Denoising Pre-training for Neural Machine Translation) can be used as the parameter initialization of the sequence decoder, thereby reducing the learning difficulty.

[0057] In this embodiment, by extracting the graphic feature sequence corresponding to the document image and the text feature sequence determined by the position and size parameters of each paragraph detection box in the document, the image feature sequence and the text feature sequence are decoded to obtain the serial number corresponding to each paragraph in the document to determine the reading order of the document. End-to-end reading order detection is achieved without the need to identify the semantics of each paragraph, which improves the speed of determining the reading order while ensuring the effect of document reading order detection.

[0058] It should be noted that in the present disclosure, a network structure framework including a visual encoder and a Transformer-based decoder model can be proposed to achieve end-to-end document detection. The end-to-end document detection model can be as follows: Figure 2 As shown, Figure 2 The figure is a flowchart of an end-to-end document detection model for detecting the reading order of each paragraph in a document.

[0059] like Figure 2As shown, the document image is first cut into blocks to obtain multiple image blocks 21, and then the multiple image blocks 21 are input into an encoder for encoding to obtain encoded image features 22. And paragraph detection is performed on the document to determine the detection box corresponding to each paragraph in the document. Figure 2 It can be seen that different paragraphs can be distinguished by detection boxes of different sizes and colors. Then, the detection boxes of each paragraph can be input with unordered text to determine the embedding vector 23 corresponding to each detection box. After that, the obtained encoded image features 22 and the embedding vector 23 corresponding to the detection box are input into the decoder for fusion decoding to obtain the sorted text output 24, so as to obtain the reading order detection result. The reading order detection result can be as follows Figure 2 As shown in the rightmost image, the serial number in each detection box indicates the order of each paragraph when reading the document.

[0060] In the disclosed embodiment, the model can also be trained to improve the robustness of the model for document reading order detection. Specifically, the encoded detection frame can be embedded into the feature and passed through the last linear layer of the model to output the classification results of each detection frame. In the training stage, each detection frame can be supervised by the sorted ID, and the training model can be supervised by calculating the cross entropy loss.

[0061] Figure 3 It is a flowchart of an end-to-end document detection method proposed in another embodiment of the present disclosure.

[0062] like Figure 3 As shown, the end-to-end document detection method includes:

[0063] S301: Based on a preset sliding window, the image corresponding to the target document is segmented to obtain an image block sequence.

[0064] The preset sliding window can be set with a suitable window size and step size according to the actual image segmentation accuracy required. The window size determines the size of the image block obtained each time, and the step size determines the distance the window moves on the image each time, that is, the distance between the image blocks obtained from two adjacent segmentations.

[0065] In the disclosed embodiment, the window can be moved on the image according to a preset step size, and at each window position, the image data within the window is extracted as the segmented image block. The above operation is repeated until the image corresponding to the target document is segmented to obtain an image block sequence.

[0066] S302: Encode the image block sequence to obtain an image feature sequence.

[0067] In the disclosed embodiment, an encoder may be used to encode the image block sequence obtained by segmentation to obtain an image feature sequence.

[0068] In this embodiment, by using a sliding window to segment the image corresponding to the target document, and encoding the segmented image block sequence to obtain an image feature sequence, the encoding efficiency of high-resolution images can be improved, thereby increasing the speed of determining the document image reading order.

[0069] S303: Perform paragraph detection on the target document to determine the position and size parameters of the detection box corresponding to each paragraph.

[0070] S304: Determine a text feature sequence corresponding to the target document according to the position and size parameters of each detection frame.

[0071] S305: fusing and decoding the image feature sequence and the text feature sequence to obtain the reading order of the target document.

[0072] The description of S303 to S305 can be specifically referred to the above embodiment, which will not be repeated here.

[0073] Figure 4 It is a flowchart of an end-to-end document detection method proposed in another embodiment of the present disclosure.

[0074] like Figure 4 As shown, the end-to-end document detection method includes:

[0075] S401: Extract features of the image corresponding to the target document to obtain an image feature sequence.

[0076] S402: Perform paragraph detection on the target document to determine the position and size parameters of the detection box corresponding to each paragraph.

[0077] S403: Determine a text feature sequence corresponding to the target document according to the position and size parameters of each detection frame.

[0078] The description of S401 to S403 can be specifically referred to the above embodiment, which will not be repeated here.

[0079] S404: Using the text feature sequence as a query vector and the image feature sequence as a key and value vector, the image feature sequence and the text feature sequence are fused to obtain fused features.

[0080] In the disclosed embodiment, the text feature sequence can be used as the query vector, the image feature sequence can be used as the key and value vectors, and the image feature sequence and the text feature sequence can be fused through the attention mechanism to obtain the fused features. Specifically, for each query vector, the similarity score with the key vector can be calculated, and then the similarity score can be converted into a probability distribution, so that each query vector in the text feature sequence will correspond to a weight distribution, indicating the degree of association between it and different key vectors in the image feature sequence. Afterwards, the value vector can be weighted and summed using the weight distribution to obtain the fused feature vector corresponding to each query vector, and the fused feature vector corresponding to each query vector in the text feature sequence can be spliced ​​to obtain the image feature sequence and the fused features of the text feature sequence corresponding to the target document.

[0081] S405: Decode the fused features to obtain the serial number corresponding to each paragraph in the target document.

[0082] In the disclosed embodiment, a decoder may be used to decode the fused features to obtain a sequence number corresponding to each paragraph in the target document.

[0083] It should be noted that in order to avoid the situation where the same paragraph is repeatedly numbered and corresponds to two different serial numbers when outputting the serial number corresponding to each paragraph, in the embodiment of the present disclosure, the selection probability value of the paragraph whose serial number has been output can be modified.

[0084] Optionally, the fused features may be decoded, and when a first serial number corresponding to any detection frame is determined, a first selection probability corresponding to the first serial number is set to a first value.

[0085] The first sequence number refers to the sequence number of the current sequence output, which may be any sequence number. The first value may be 0 or a very small value to ensure that when determining the next sequence detection frame, the probability of the already determined detection frame being selected is very small, and repeated reading will not occur.

[0086] In the disclosed embodiment, the model can output the sorting results of each detection box one by one by autoregression. When the first serial number corresponding to any detection box is determined, the any detection box is a detection box that has been output and can be shielded with a lower probability (i.e., the first value) to avoid subsequent output of repeated detection boxes.

[0087] In this embodiment, the text feature sequence is used as the query vector and the image feature sequence is used as the key and value vector to complete the fusion of the image feature sequence and the text feature sequence, and then the fused features are encoded to obtain the sequence number corresponding to each paragraph in the target document. Since the fused features contain information of both image and text modalities, they can more comprehensively reflect the characteristics of the target document, improve the accuracy and reliability of the sequence number obtained after decoding, and improve the robustness of determining the document reading order.

[0088] Figure 5 It is a flowchart of an end-to-end document detection method proposed in another embodiment of the present disclosure.

[0089] like Figure 5 As shown, the end-to-end document detection method includes:

[0090] S501: Extract features of the image corresponding to the target document to obtain an image feature sequence.

[0091] S502: Perform paragraph detection on the target document to determine the position and size parameters of the detection box corresponding to each paragraph.

[0092] S503: Determine a text feature sequence corresponding to the target document according to the position and size parameters of each detection frame.

[0093] S504: fusing and decoding the image feature sequence and the text feature sequence to obtain the reading order of the target document.

[0094] The description of S501 to S504 can be specifically referred to the above embodiment, which will not be repeated here.

[0095] S505: Determine the display position of the serial number corresponding to each detection frame according to the position and size parameters of the detection frame corresponding to each paragraph.

[0096] In the disclosed embodiment, after obtaining the reading order of the target document, the serial number corresponding to each paragraph can be displayed in the target document according to the reading order. The position of the serial number should be related to the detection box corresponding to the paragraph in the target document. Therefore, the position and size parameters of the detection box corresponding to each paragraph can be used to determine the display position of the serial number corresponding to each detection box.

[0097] It should be noted that the display position of the serial number corresponding to each detection frame can be the beginning of a paragraph, or can also be a corner of the detection frame. For example, when the size of the detection frame is very small, the display position of the serial number can be the beginning of a paragraph.

[0098] S506: Based on the display position of the serial number corresponding to each detection frame, output the target document containing the serial number.

[0099] In the embodiment of the present disclosure, the effect diagram of the target document containing the sequence number can be as follows: Figure 6 As shown, Figure 6 This is an example of the effect of performing reading order detection on a document. Figure 6 It can be seen that different paragraphs (i.e., detection frames in the present disclosure) can be framed by multiple rectangular frames in the target document, and each detection frame can display the serial number of the paragraph at the corresponding position to determine the reading order of the target document. When the size of the detection frame is small (such as only one line), the serial number is displayed at the beginning of the paragraph. When the size of the detection frame is large, the serial number is displayed in the last line of the paragraph, i.e., the lower left corner of the detection frame.

[0100] It should be noted that, when there are multiple images corresponding to the target document, the paragraphs can be numbered in each image, that is, the paragraph numbers in each image can start from 0, and the paragraph numbers in two adjacent images are not continuous. Alternatively, the paragraph numbers in two adjacent images can also be continuous, that is, the last paragraph of the previous image is numbered 10, and the first paragraph of the next image is numbered 11. This disclosure does not limit this.

[0101] In this embodiment, the display position of the serial number corresponding to each detection box is determined according to the position and size parameters of the detection box corresponding to each paragraph, and then the target document containing the serial number is output based on the display position of the serial number corresponding to each detection box. The reading order between the paragraphs in the target document can be intuitively determined, the readability of the document is improved, and the user experience is optimized.

[0102] Optionally, after determining the serial number corresponding to each paragraph, a second color corresponding to the serial number may be determined according to the first color corresponding to the text in the target document, wherein the second color is different from the first color. Then, the target document containing the second color serial number is output.

[0103] For example, when the first color corresponding to the text in the target document is black, the second color of the serial number corresponding to each paragraph in the target document may be blue. Thus, after the target document containing the serial number is output, the user is prevented from confusing the serial number with the content in the document, and the efficiency of determining the reading order of each paragraph is improved.

[0104] Optionally, after determining the serial number corresponding to each paragraph, the distance between each two adjacent detection frames can also be determined according to the position and size parameters of each detection frame. Then, based on the distance between each two adjacent detection frames, a target document including the detection frames and the corresponding serial numbers is output, wherein the line widths corresponding to the detection frames whose distances are less than a threshold are the first width, and the line widths corresponding to the detection frames whose distances are greater than or equal to the threshold are the second width.

[0105] It should be noted that the threshold, the first width and the second width can be set based on experience to ensure the readability of the document, and the first width is smaller than the second width. When the distance between adjacent detection frames is smaller than the threshold, the line width of the detection frame can be appropriately reduced, that is, set to the first width, to ensure that the lines of two adjacent detection frames do not overlap, thereby enhancing the visual distinction between the detection frames.

[0106] In the disclosed embodiment, by adjusting the line width of the detection boxes in the target document based on the distance between adjacent detection boxes, the typesetting and layout of the detection boxes in the target document can be further optimized, the readability of the document can be improved, and the user experience can be enhanced.

[0107] Figure 7 It is a structural diagram of an end-to-end document detection device proposed in an embodiment of the present disclosure.

[0108] like Figure 7 As shown, the end-to-end document detection device 700 includes:

[0109] The encoding module 701 is used to extract features of the image corresponding to the target document to obtain an image feature sequence;

[0110] The first determination module 702 is used to perform paragraph detection on the target document and determine the position and size parameters of the detection box corresponding to each paragraph;

[0111] The second determination module 703 is used to determine the text feature sequence corresponding to the target document according to the position and size parameters of each detection frame;

[0112] The decoding module 704 is used to fuse and decode the image feature sequence and the text feature sequence to obtain the reading order of the target document.

[0113] In some possible embodiments, the encoding module 701 may be specifically used for:

[0114] Based on the preset sliding window, the image corresponding to the target document is segmented to obtain an image block sequence;

[0115] The image block sequence is encoded to obtain an image feature sequence.

[0116] In some possible embodiments, the decoding module 704 may be specifically configured to:

[0117] Take the text feature sequence as the query vector and the image feature sequence as the key and value vector, fuse the image feature sequence and the text feature sequence to obtain the fused features;

[0118] Decode the fused features to obtain the serial number corresponding to each paragraph in the target document.

[0119] In some possible embodiments, the decoding module 704 may be specifically configured to:

[0120] The fused features are decoded, and when the first serial number corresponding to any detection frame is determined, the first selection probability corresponding to the first serial number is set to a first value.

[0121] In some possible embodiments, the decoding module 704 may also be used to:

[0122] Determine the display position of the serial number corresponding to each detection frame according to the position and size parameters of the detection frame corresponding to each paragraph;

[0123] Based on the display position of the serial number corresponding to each detection box, the target document containing the serial number is output.

[0124] In some possible embodiments, the decoding module 704 may also be used to:

[0125] Determine a second color corresponding to the serial number according to a first color corresponding to the text in the target document, wherein the second color is different from the first color;

[0126] Output the target document containing the second color sequence number.

[0127] In some possible embodiments, the decoding module 704 may also be used to:

[0128] According to the position and size parameters of each detection frame, determine the distance between every two adjacent detection frames;

[0129] Based on the distance between every two adjacent detection frames, a target document containing detection frames and corresponding serial numbers is output, wherein the line widths corresponding to the detection frames whose distances are less than a threshold are the first width, and the line widths corresponding to the detection frames whose distances are greater than or equal to the threshold are the second width.

[0130] It should be noted that the above explanation of the end-to-end document detection method is also applicable to the end-to-end document detection device of this embodiment, and will not be repeated here.

[0131] In this embodiment, by extracting the graphic feature sequence corresponding to the document image and the text feature sequence determined by the position and size parameters of each paragraph detection box in the document, the image feature sequence and the text feature sequence are decoded to obtain the serial number corresponding to each paragraph in the document to determine the reading order of the document. End-to-end reading order detection is achieved without the need to identify the semantics of each paragraph, which improves the speed of determining the reading order while ensuring the effect of document reading order detection.

[0132] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0133] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0134] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0135] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0136] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as an end-to-end document detection method. For example, in some embodiments, the end-to-end document detection method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the end-to-end document detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the end-to-end document detection method in any other appropriate manner (eg, by means of firmware).

[0137] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0139] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0141] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0142] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0143] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0144] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the description of the present disclosure, the words "if" and "if" used can be interpreted as "at the time of" or "when" or "in response to determination" or "under the circumstances of".

[0145] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An end-to-end document detection method, comprising: Extract features of the image corresponding to the target document to obtain an image feature sequence; Perform paragraph detection on the target document to determine the position and size parameters of the detection box corresponding to each paragraph; Determining a text feature sequence corresponding to the target document according to the position and size parameters of each detection frame; The image feature sequence and the text feature sequence are fused and decoded to obtain the reading order of the target document.

2. The method of claim 1, wherein: The step of extracting features from the image corresponding to the target document to obtain an image feature sequence includes: Based on a preset sliding window, the image corresponding to the target document is segmented to obtain an image block sequence; The image block sequence is encoded to obtain the image feature sequence.

3. The method of claim 1, wherein: The step of fusing and decoding the image feature sequence and the text feature sequence to obtain the reading order of the target document includes: Using the text feature sequence as a query vector and the image feature sequence as a key and value vector, the image feature sequence and the text feature sequence are fused to obtain fused features; The fused features are decoded to obtain a sequence number corresponding to each paragraph in the target document.

4. The method of claim 3, wherein: Decoding the fused features to obtain a sequence number corresponding to each paragraph in the target document includes: The fused features are decoded, and when a first serial number corresponding to any detection frame is determined, a first selection probability corresponding to the first serial number is set to a first value.

5. The method according to any one of claims 1 to 4, wherein: After fusing and decoding the image feature sequence and the text feature sequence to obtain the reading order of the target document, the method further includes: Determining the display position of the serial number corresponding to each detection frame according to the position and size parameters of the detection frame corresponding to each paragraph; Based on the display position of the serial number corresponding to each detection frame, a target document containing the serial number is output.

6. The method of claim 5, wherein: After fusing and decoding the image feature sequence and the text feature sequence to obtain the reading order of the target document, the method further includes: Determining a second color corresponding to the serial number according to a first color corresponding to the text in the target document, wherein the second color is different from the first color; The target document containing the second color sequence number is output.

7. The method of claim 5, wherein: After obtaining the serial number corresponding to each paragraph in the target document, the method further includes: Determine the distance between every two adjacent detection frames according to the position and size parameters of each detection frame; Based on the distance between every two adjacent detection frames, a target document including the detection frames and corresponding serial numbers is output, wherein the line widths corresponding to the detection frames whose distances are less than a threshold are the first width, and the line widths corresponding to the detection frames whose distances are greater than or equal to the threshold are the second width.

8. An end-to-end document detection device, comprising: The encoding module is used to extract features of the image corresponding to the target document to obtain an image feature sequence; A first determination module is used to perform paragraph detection on the target document and determine the position and size parameters of the detection box corresponding to each paragraph; A second determination module is used to determine the text feature sequence corresponding to the target document according to the position and size parameters of each detection frame; The decoding module is used to fuse and decode the image feature sequence and the text feature sequence to obtain the reading order of the target document.

9. The device of claim 8, wherein: The encoding module is specifically used for: Based on a preset sliding window, the image corresponding to the target document is segmented to obtain an image block sequence; The image block sequence is encoded to obtain the image feature sequence.

10. The device of claim 9, wherein: The decoding module is specifically used for: Using the text feature sequence as a query vector and the image feature sequence as a key and value vector, the image feature sequence and the text feature sequence are fused to obtain fused features; The fused features are decoded to obtain a sequence number corresponding to each paragraph in the target document.

11. The device of claim 10, wherein: The decoding module is specifically used for: The fused features are decoded, and when a first serial number corresponding to any detection frame is determined, a first selection probability corresponding to the first serial number is set to a first value.

12. The device according to any one of claims 8 to 11, wherein: The decoding module is further used for: Determining the display position of the serial number corresponding to each detection frame according to the position and size parameters of the detection frame corresponding to each paragraph; Based on the display position of the serial number corresponding to each detection frame, a target document containing the serial number is output.

13. The device of claim 12, wherein: The decoding module is further used for: Determining a second color corresponding to the serial number according to a first color corresponding to the text in the target document, wherein the second color is different from the first color; The target document containing the second color sequence number is output.

14. The device of claim 12, wherein: The decoding module is further used for: Determine the distance between every two adjacent detection frames according to the position and size parameters of each detection frame; Based on the distance between every two adjacent detection frames, a target document including the detection frames and corresponding serial numbers is output, wherein the line widths corresponding to the detection frames whose distances are less than a threshold are the first width, and the line widths corresponding to the detection frames whose distances are greater than or equal to the threshold are the second width.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the end-to-end document detection method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to enable the computer to execute the end-to-end document detection method according to any one of claims 1 to 7.

17. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the end-to-end document detection method according to any one of claims 1 to 7.

Images