Image processing method, device and system, electronic equipment, storage medium and product

Abstract

The embodiment of the invention provides an image processing method, device and system, electronic equipment, a storage medium and a product. The method comprises the following steps: determining each anchor frame feature data in a to-be-detected document image through a query module in a trained image processing model, and determining each textbox coordinate of the to-be-detected document image through an OCR (Optical Character Recognition) module of an image detection model, inputting each anchor frame feature data and each textbox coordinate in the to-be-detected document image into a fusion module of the image detection model to generate each enhanced anchor frame coordinate of the to-be-detected document image, and determining local feature data corresponding to each enhanced anchor frame coordinate through a local decoding module of the image detection model, therefore, layout images and categories corresponding to the enhanced anchor frames are obtained. The method is used for achieving the effects of improving the detection efficiency and accuracy of the document layout and improving the business handling efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image processing method, device, system, electronic device, storage medium and product. Background Art

[0002] In the era of digital finance, banks and financial institutions need to process massive amounts of documents. With the surge in business volume, traditional manual document processing methods can no longer meet the requirements of efficiency and accuracy.

[0003] Currently, the existing technology uses traditional image recognition technology to identify documents and obtain category information of each layout area in the document, such as title, text, table, picture, etc., and then identifies each layout area based on different category information to obtain information in each layout area.

[0004] However, since the document layout usually has certain format specifications and / or compliance requirements, it is difficult for the existing technology to accurately detect the layout information of each document, which reduces the detection efficiency of the document layout and further reduces the business processing efficiency. Summary of the invention

[0005] The embodiments of the present application provide image processing methods, devices, systems, electronic devices, storage media and products for improving the efficiency and accuracy of document layout detection and improving business processing efficiency.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0007] Acquire the document image to be detected;

[0008] The document image to be detected is processed by the trained image detection model to obtain the layout data of each document image to be detected; the image detection model includes a query module, an OCR module, a fusion module and a local decoding module; the layout data includes a layout image and a corresponding category;

[0009] Among them, the query module is used to determine the feature data of each anchor frame in the document image to be detected, the OCR module is used to determine the coordinates of each text frame of the document image to be detected, and the fusion module is used to generate the coordinates of each enhanced anchor frame of the document image to be detected according to the anchor frame feature data and text frame coordinates of the document image to be detected; the local decoding module is used to determine the local feature data of the enhanced anchor frame based on the local range corresponding to each enhanced anchor frame coordinate, and obtain the layout data corresponding to each enhanced anchor frame.

[0010] Optionally, generating the coordinates of each enhanced anchor frame of the document image to be detected according to the coordinates of the anchor frame and the text frame of the document image to be detected specifically includes:

[0011] Obtain the coordinates of each text box of the document image to be detected, and perform feature embedding processing to obtain feature data of each text box;

[0012] By determining each text box feature data and each anchor box feature data in the document image to be detected and performing mean summation processing, each enhanced anchor box coordinate of the document image to be detected is obtained.

[0013] Optionally, the image detection model further includes a feature extraction module; the feature extraction module is used to extract features of the document image to be detected to obtain feature data of the document image to be detected; wherein the feature extraction module includes a backbone network module and an encoding module; and further includes:

[0014] Input the feature data into the local decoding module;

[0015] The feature data and the coordinates of each strong anchor frame are processed by the local decoding module based on the local attention mechanism to obtain the local feature data of each enhanced anchor frame; the layout data corresponding to the enhanced anchor frame is determined according to the local feature data.

[0016] Optionally, determining feature data of each anchor box in the document image to be detected specifically includes:

[0017] Determine the confidence threshold corresponding to the document image to be detected according to the preset number;

[0018] Determine feature data of each anchor frame in the document image to be detected according to a confidence threshold; wherein the number of anchor frames in the document image to be detected does not exceed a preset number.

[0019] Optionally, before acquiring the image of the document to be detected, the method includes:

[0020] Obtain a training data set, the training data set including a plurality of training document images and coordinates of each text box corresponding to each training document image;

[0021] The training model is iteratively trained using the training data set to obtain a trained image detection model;

[0022] Among them, the training document image is obtained by performing image restoration processing on the document source image, the number of layout image categories corresponding to the training data set is not less than the preset number of categories, and the difference in the number of layout images of different categories does not exceed the preset difference.

[0023] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0024] An acquisition module, used for acquiring the image of the document to be detected;

[0025] A processing module is used to process the document image to be detected through the trained image detection model to obtain the layout data of each document image to be detected; the image detection model includes a query module, an OCR module, a fusion module and a local decoding module; the layout data includes a layout image and a corresponding category;

[0026] Among them, the query module is used to determine the feature data of each anchor frame in the document image to be detected, the OCR module is used to determine the coordinates of each text frame of the document image to be detected, and the fusion module is used to generate the coordinates of each enhanced anchor frame of the document image to be detected according to the anchor frame feature data and text frame coordinates of the document image to be detected; the local decoding module is used to determine the local feature data of the enhanced anchor frame based on the local range corresponding to each enhanced anchor frame coordinate, and obtain the layout data corresponding to each enhanced anchor frame.

[0027] Optionally, the processing module is further used to obtain the coordinates of each text box of the document image to be detected, and perform feature embedding processing to obtain feature data of each text box;

[0028] By determining each text box feature data and each anchor box feature data in the document image to be detected and performing mean summation processing, each enhanced anchor box coordinate of the document image to be detected is obtained.

[0029] Optionally, the image detection model further includes a feature extraction module; the feature extraction module is used to extract features of the document image to be detected to obtain feature data of the document image to be detected; wherein the feature extraction module includes a backbone network module and an encoding module;

[0030] The processing module is also used to input the feature data into the local decoding module;

[0031] The feature data and the coordinates of each strong anchor frame are processed by the local decoding module based on the local attention mechanism to obtain the local feature data of each enhanced anchor frame; the layout data corresponding to the enhanced anchor frame is determined according to the local feature data.

[0032] Optionally, the processing module is further used to determine a confidence threshold corresponding to the document image to be detected according to a preset number;

[0033] Determine feature data of each anchor frame in the document image to be detected according to a confidence threshold; wherein the number of anchor frames in the document image to be detected does not exceed a preset number.

[0034] Optionally, the processing module is further used to obtain a training data set before obtaining the document image to be detected, the training data set including a plurality of training document images and coordinates of each text box corresponding to each training document image;

[0035] The training model is iteratively trained using the training data set to obtain a trained image detection model;

[0036] Among them, the training document image is obtained by performing image restoration processing on the document source image, the number of layout image categories corresponding to the training data set is not less than the preset number of categories, and the difference in the number of layout images of different categories does not exceed the preset difference.

[0037] In a third aspect, an embodiment of the present application provides a business data system, including:

[0038] Document analysis unit and layout recognition unit;

[0039] The document analysis unit is used to execute the first aspect and / or various possible implementations of the first aspect as described above;

[0040] The layout recognition unit is used to receive layout images of various categories output by the document analysis unit, and determine information in the layout images based on the categories of the layout images.

[0041] In a fourth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0042] Memory stores computer-executable instructions;

[0043] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0044] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0045] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0046] The image processing method, device, system, electronic device, storage medium and product provided by the embodiments of the present application obtain a document image to be detected and input it into a trained image detection model, determine the feature data of each anchor frame in the document image to be detected through the query module in the image processing model, and determine the coordinates of each text frame in the document image to be detected through the OCR module of the image detection model, input the feature data of each anchor frame and the coordinates of each text frame in the document image to be detected into the fusion module in the image processing model, generate the coordinates of each enhanced anchor frame of the document image to be detected through the fusion module, determine the local feature data corresponding to each enhanced anchor frame coordinate through the local decoding module of the image detection model, and obtain the layout image and category corresponding to each enhanced anchor frame. The present application improves the detection efficiency and accuracy of the document layout, and further improves the efficiency of business handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 A schematic diagram of the application scenario provided for this application;

[0049] Figure 2 Schematic diagram of the image processing method provided in this application Figure 1 ;

[0050] Figure 3 Schematic diagram of the image processing method provided in this application Figure 2 ;

[0051] Figure 4 A schematic diagram of the structure of the image processing device provided by this application;

[0052] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0053] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0054] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0056] First, the terms involved in this application are explained:

[0057] OCR (Optical Character Recognition): refers to a technology that analyzes text in an image and converts it into editable digital text.

[0058] In the era of digital finance, banks and financial institutions are faced with the need to process massive amounts of documents, including but not limited to customer application forms, transaction records, agreements, etc. With the surge in business volume, traditional manual document processing methods can no longer meet the requirements of efficiency and accuracy.

[0059] With the development of computer vision and natural language processing technology, object detection technology has made significant progress. Therefore, the existing technology establishes a business data system based on the object detection technology and processes massive documents through the business data system.

[0060] Figure 1 A schematic diagram of the application scenario provided for this application, such as Figure 1As shown, the specific application scenario of the present application includes a business data system, which includes a document analysis unit and a layout recognition unit. The prior art establishes a document analysis unit based on a traditional target detection model, establishes a layout recognition unit based on data extraction technology, and generates a business data system. The document image to be detected is input into the business data system, and the document image to be detected is processed by the document analysis unit to obtain each target frame existing in the document image to be detected, and each target frame is identified by the layout recognition unit, and the key information in the layout image corresponding to each target frame is extracted, and the key information is sent to the terminal device to meet the needs of the staff and improve the efficiency of business processing. Among them, the document includes layout images of different categories, including but not limited to titles, texts, tables, pictures, etc.

[0061] However, there are differences in the characteristics of documents and photos. For example, there are more texts in documents, the target boxes are slender in shape, fewer in number, and tend to appear in clusters. The size variance of different document images to be detected is large, and the correlation between adjacent target boxes is high and the probability of mutual occlusion is high. In addition, there are many types of documents in banking and financial scenarios (such as checks, reports, and application forms), and each document has its own specific format and layout. However, the target detection model of the existing technology is difficult to accurately identify or process documents of various formats, which reduces the accuracy of layout segmentation, resulting in large errors in the extracted layout information, further affecting the efficiency of business processing.

[0062] The image processing method provided by the present application is to iteratively train the training model through the training data set to obtain the trained image detection model, input the document image to be detected into the trained image detection model, process the document image to be detected through the feature extraction module in the image detection model to obtain the feature data in the document image to be detected, generate the feature data of each anchor frame in the document image to be detected according to the feature data through the query module, and process the document image to be detected through the OCR module to obtain the coordinates of each text frame, thereby fusing the feature data of each anchor frame and the text frame coordinates through the fusion module to obtain the enhanced anchor frame coordinates, input the enhanced anchor frame coordinates and the feature data into the local decoding module, determine the feature data around each enhanced anchor frame coordinate through the local decoding module to obtain the corresponding layout image and the category of the layout image and output it, so that the layout recognition unit determines the information in the corresponding layout image according to the category of each layout image. Improve the accuracy and efficiency of document analysis, improve the recognition results of layout images, and further improve the efficiency of business handling.

[0063] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0064] Figure 2 Schematic diagram of the image processing method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0065] S201, obtaining a document image to be detected.

[0066] More specifically, the document image to be detected is an image corresponding to a document that needs to be analyzed (or a document that needs to have data extracted).

[0067] Exemplarily, the document image to be detected may be a screenshot of an electronic document, a photographed image of a paper document, or a scanned image of a paper document. The present application does not limit the source of the document image to be detected.

[0068] Exemplarily, for a distorted captured image, a repair process or a document restoration process is performed on the captured image to obtain a document image to be detected corresponding to the captured image.

[0069] S202. Process the document image to be detected by the trained image detection model to obtain the layout data of each document image to be detected; the image detection model includes a query module, an OCR module, a fusion module and a local decoding module.

[0070] More specifically, the document image to be detected is processed by the trained image detection model to obtain the layout data of each document image to be detected; the image detection model includes a query module, an OCR module, a fusion module and a local decoding module; the layout data includes a layout image and a corresponding category; wherein the query module is used to determine the feature data of each anchor frame in the document image to be detected, the OCR module is used to determine the coordinates of each text frame of the document image to be detected, and the fusion module is used to generate the coordinates of each enhanced anchor frame of the document image to be detected according to the feature data of the anchor frame and the coordinates of the text frame of the document image to be detected; the local decoding module is used to determine the local feature data of the enhanced anchor frame based on the local range corresponding to each enhanced anchor frame coordinate, and obtain the layout data corresponding to each enhanced anchor frame. The anchor frame feature data includes but is not limited to the position, size and shape information of the anchor frame in the document image.

[0071] Optionally, determining each anchor frame feature data in the document image to be detected specifically includes: determining a confidence threshold corresponding to the document image to be detected according to a preset number; determining each anchor frame feature data in the document image to be detected according to the confidence threshold; wherein the number of anchor frames in the document image to be detected does not exceed a preset number.

[0072] This embodiment determines the confidence threshold of the current document image match based on a preset number, thereby improving the robustness of the image processing model in detecting documents with large differences in target quantity. While ensuring the accuracy of the layout data, it improves the detection rate and further improves business processing efficiency.

[0073] Optionally, the coordinates of each enhanced anchor frame of the document image to be detected are generated according to the coordinates of the anchor frame and text frame of the document image to be detected, specifically including: obtaining the coordinates of each text frame of the document image to be detected, and performing feature embedding processing to obtain feature data of each text frame; determining the feature data of each text frame and each anchor frame in the document image to be detected, performing mean summation processing, and obtaining the coordinates of each enhanced anchor frame of the document image to be detected.

[0074] Optionally, the feature embedding processing and mean summation processing methods in this embodiment are similar to the implementation principles and technical effects of the disclosed technologies, and are not described in detail in this embodiment.

[0075] In a possible embodiment, the coordinates of each text box of the document image to be detected are converted into corresponding feature vectors by a fusion module to obtain the embedded feature data of each text box. The matrix dimensions of the text box feature data corresponding to the document image to be detected and the corresponding anchor box feature data are the same.

[0076] Exemplarily, the text box feature data corresponding to the document image to be detected is a feature vector containing four elements, and the corresponding anchor box feature data is a feature vector containing four elements. The feature vector of the text box feature data and the feature vector of the anchor box feature data are added element by element, and the average value is taken based on the number of elements of the feature vector to obtain the corresponding new enhanced feature vector, which is the enhanced anchor box coordinates of the document image to be detected.

[0077] This embodiment fuses the text box coordinates generated by the OCR module with the anchor box feature data output by the query module to enhance the understanding and processing capabilities of the anchor box output by the query module on the content of the document image to be detected, improve the adaptability of the image detection model to the format and content of complex documents, enhance the capture effect of the structure and content in the document image to be detected, and improve the accuracy of document analysis and understanding.

[0078] Optionally, the image detection model also includes a feature extraction module; the feature extraction module is used to extract features of the document image to be detected to obtain feature data of the document image to be detected; wherein the feature extraction module includes a backbone network module and an encoding module.

[0079] Exemplarily, the backbone network module is used to extract features from the document image to be detected to obtain multi-scale feature data in the document image to be detected, and the multi-scale feature data is processed by the encoding module to obtain deep feature data.

[0080] Optionally, the feature data is input into a local decoding module; the feature data and the coordinates of each strong anchor frame are processed by the local decoding module based on a local attention mechanism to obtain local feature data of each enhanced anchor frame; and the layout data corresponding to the enhanced anchor frame is determined based on the local feature data.

[0081] In a possible embodiment, for any enhanced anchor frame coordinate, local feature data of an area adjacent to the coordinate is determined by a local decoding module, and the corresponding layout image and the category of the layout image are determined based on the local feature data.

[0082] Exemplarily, when the local feature data indicates page number information, it is determined that the layout image category corresponding to the local area near the coordinates of the enhanced anchor frame is a footer.

[0083] Exemplarily, when the local feature data corresponding to the enhanced anchor frame coordinates indicates text information, the layout image category corresponding to the local area near the enhanced anchor frame coordinates is determined to be text, and when the local feature data corresponding to the next enhanced anchor frame coordinates also indicates text information, the layout image category corresponding to the local area near the next enhanced anchor frame coordinates is determined to be text, and two layout images of the text category are obtained. This embodiment realizes effective detection and segmentation of the layout of continuous long text paragraphs by determining the local feature data of the enhanced anchor frame coordinates, improves the detection rate and accuracy of the layout image, reduces the detection complexity, improves the document analysis efficiency, and further improves the business processing efficiency.

[0084] Optionally, before obtaining the document image to be detected, a training data set is obtained, the training data set including multiple training document images and the coordinates of each text box corresponding to each training document image; the training data set is used to iteratively train the training model to obtain a trained image detection model; wherein, the training document image is obtained by performing image restoration processing on the document source image, the number of layout image categories corresponding to the training data set is not less than the preset number of categories, and the difference in the number of layout images of different categories does not exceed the preset difference.

[0085] In a possible embodiment, the training model is trained 300 times with a learning rate of 0.0001 using an AdamW optimizer.

[0086] Optionally, after the iterative training of the training model is completed, the AP (Average Precision) under different IOUs (Intersection over Union, a measurement standard) is used to calculate the accuracy of the image detection model; GFLOPS (Giga Floating Point Operations Per Second, floating point operations) and FPS (Frames Per Second, frames per second) are used to calculate the calculation rate of the image detection model. When the accuracy and calculation rate of the image detection model reach the preset indicator values, the image detection model is obtained. The method for calculating the average accuracy, the number of floating point operations and the number of frames per second in this embodiment is similar to the implementation principle and technical effect of the prior art, and this embodiment will not be repeated here.

[0087] Exemplarily, layout image categories include but are not limited to: tables, line graphs, histograms, pie charts, texts, abstracts, citations, seals, etc. The number of categories included in the training data set is not less than the preset number of categories, and the difference in the number of images in different categories does not exceed the preset difference.

[0088] In a possible embodiment, if there is a distorted training document image, the image is repaired using a document restoration method and added to the training data set after repair. Wherein, each training document image in the training data set is an image in coco format.

[0089] This embodiment improves the stability and reliability of the image detection model in processing diverse documents and improves the generalization of the image detection model for document analysis by adding diverse document image training data sets to iteratively train the image detection model.

[0090] The image processing method provided in the embodiment of the present application determines the feature data of each anchor frame in the document image to be detected through an image detection model, and determines the coordinates of each text frame of the document image to be detected, thereby obtaining enhanced anchor frame coordinates suitable for the document image to be detected, and determines the corresponding layout image and category by determining the local feature data near each enhanced anchor frame coordinate, thereby improving the detection accuracy of the document image, improving the efficiency of document image processing, and further improving the business handling efficiency.

[0091] Figure 3 Schematic diagram of the image processing method provided in this application Figure 2 ,like Figure 3 As shown, the image detection model includes a backbone network module, an encoding module, a query module, an OCR module, a fusion module and a local decoding module.

[0092] In a possible embodiment, the document image to be detected is input into the image detection model, the document image to be detected is processed by the backbone network module (such as the backbone neural network), the multi-scale feature data in the document image to be detected is extracted, the multi-scale feature data is input into the encoding module, and the multi-scale feature data extracted by the backbone neural network is further processed by the encoding module to capture more complex patterns and relationships (such as context information integration, feature compression, feature expansion), and obtain the feature data of the document image to be detected. The connection between the backbone network module and the encoding module effectively improves the feature extraction and representation capabilities of the image detection model, thereby further improving the overall performance of the image detection model. Among them, the backbone neural network of this embodiment is generated based on FPN (Feature Pyramid Networks).

[0093] In a possible embodiment, the feature data of the document image to be detected is input into the query module, and the query module generates no more than a preset number of anchor frames according to the feature data, and obtains the corresponding feature data of each anchor frame. The query module of this embodiment is generated based on FFN (Feedforward Neural Network).

[0094] In a possible embodiment, the document image to be detected is processed by an OCR module (e.g., an OCR engine) to obtain image layout information (e.g., coordinates of each text box) in the document image to be detected. The coordinates of each text box and the feature data of each anchor box are input into a fusion module, and the anchor box feature data and the image layout information in the document image to be detected are fused by the fusion module to obtain enhanced anchor box coordinates, that is, enhanced target query results. The coordinates and feature data of each enhanced anchor box of the document image to be detected are input into a local decoding module, and the local feature data of the adjacent area corresponding to the position of the enhancement module coordinates is determined by the local decoding module, so as to obtain the layout data corresponding to each adjacent area according to the local feature data. Among them, the local decoding module is generated based on the local attention mechanism.

[0095] The image processing method provided in the embodiment of the present application, by fusing the anchor frame feature data and the graphic layout information, and determining multiple local feature data in the image through a local decoding module, further determines each layout image in the document image and the corresponding category information, thereby enhancing the adaptability of the image processing model to various document formats and layouts, improving the overall performance of the image detection model, and improving the detection efficiency and accuracy of the document layout.

[0096] Alternatively, if Figure 1As shown, the business data system includes a document analysis unit and a layout recognition unit. The document analysis unit includes an image detection model, which is used to process the document data to be detected, obtain layout images of various categories and output them to the layout recognition unit. The layout recognition unit is used to receive layout images of various categories output by the document analysis unit, and determine the information in the layout image based on the category of the layout image.

[0097] Exemplarily, the layout recognition unit includes recognition modules corresponding to a plurality of categories, and the recognition module of each category is established according to the recognition mechanism corresponding to the category.

[0098] In a possible embodiment, the layout recognition unit determines the recognition module corresponding to the header type according to the received layout image of the header category, and inputs the layout image into the recognition module, and the recognition module performs recognition processing on the layout image of the same category to obtain key information in the layout image and output it. By using the recognition unit of the same category to recognize the layout image, the accuracy of the recognition result is improved.

[0099] Figure 4 A schematic diagram of the structure of the image processing device provided in this application, such as Figure 4 As shown, the image processing device 40 provided in this embodiment includes:

[0100] The acquisition module 401 is used to acquire the image of the document to be detected;

[0101] Processing module 402, used to process the document image to be detected by using the trained image detection model to obtain the layout data of each document image to be detected; the image detection model includes a query module, an OCR module, a fusion module and a local decoding module; the layout data includes a layout image and a corresponding category;

[0102] Among them, the query module is used to determine the feature data of each anchor frame in the document image to be detected, the OCR module is used to determine the coordinates of each text frame of the document image to be detected, and the fusion module is used to generate the coordinates of each enhanced anchor frame of the document image to be detected according to the anchor frame feature data and text frame coordinates of the document image to be detected; the local decoding module is used to determine the local feature data of the enhanced anchor frame based on the local range corresponding to each enhanced anchor frame coordinate, and obtain the layout data corresponding to each enhanced anchor frame.

[0103] Optionally, the processing module 402 is further used to obtain the coordinates of each text box of the document image to be detected, and perform feature embedding processing to obtain feature data of each text box;

[0104] By determining each text box feature data and each anchor box feature data in the document image to be detected and performing mean summation processing, each enhanced anchor box coordinate of the document image to be detected is obtained.

[0105] Optionally, the image detection model further includes a feature extraction module; the feature extraction module is used to extract features of the document image to be detected to obtain feature data of the document image to be detected; wherein the feature extraction module includes a backbone network module and an encoding module;

[0106] The processing module 402 is also used to input the feature data into the local decoding module;

[0107] The feature data and the coordinates of each strong anchor frame are processed by the local decoding module based on the local attention mechanism to obtain the local feature data of each enhanced anchor frame; the layout data corresponding to the enhanced anchor frame is determined according to the local feature data.

[0108] Optionally, the processing module 402 is further configured to determine a confidence threshold corresponding to the document image to be detected according to a preset number;

[0109] Determine feature data of each anchor frame in the document image to be detected according to a confidence threshold; wherein the number of anchor frames in the document image to be detected does not exceed a preset number.

[0110] Optionally, the processing module 402 is further configured to obtain a training data set before obtaining the document image to be detected, the training data set including a plurality of training document images and coordinates of each text box corresponding to each training document image;

[0111] The training model is iteratively trained using the training data set to obtain a trained image detection model;

[0112] Among them, the training document image is obtained by performing image restoration processing on the document source image, the number of layout image categories corresponding to the training data set is not less than the preset number of categories, and the difference in the number of layout images of different categories does not exceed the preset difference.

[0113] The image processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0114] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0115] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0116] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0117] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0118] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0119] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0120] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0121] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0122] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0123] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0124] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0127] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0128] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0129] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire the document image to be detected; The document image to be detected is processed by the trained image detection model to obtain various layout data in the document image to be detected; the image detection model includes a query module, an OCR module, a fusion module and a local decoding module; the layout data includes a layout image and a corresponding category; Among them, the query module is used to determine the feature data of each anchor frame in the document image to be detected, the OCR module is used to determine the coordinates of each text frame of the document image to be detected, and the fusion module is used to generate the coordinates of each enhanced anchor frame of the document image to be detected according to the anchor frame feature data and text frame coordinates of the document image to be detected; the local decoding module is used to determine the local feature data of the enhanced anchor frame based on the local range corresponding to each enhanced anchor frame coordinate, and obtain the layout data corresponding to each enhanced anchor frame.

2. The method according to claim 1, characterized in that: Generating coordinates of each enhanced anchor frame of the document image to be detected according to the anchor frame and text frame coordinates of the document image to be detected, specifically comprising: Acquire the coordinates of each text box of the document image to be detected, and perform feature embedding processing to obtain feature data of each text box; By determining each text box feature data and each anchor box feature data in the document image to be detected and performing mean summation processing, each enhanced anchor box coordinate of the document image to be detected is obtained.

3. The method according to claim 2, characterized in that The image detection model also includes a feature extraction module; the feature extraction module is used to extract features of the document image to be detected to obtain feature data of the document image to be detected; wherein the feature extraction module includes a backbone network module and an encoding module; The method further comprises: Inputting the feature data into the local decoding module; The feature data and the coordinates of each strong anchor frame are processed by the local decoding module based on the local attention mechanism to obtain the local feature data of each enhanced anchor frame; and the layout data corresponding to the enhanced anchor frame is determined according to the local feature data.

4. The method according to claim 1, characterized in that: Determining the feature data of each anchor frame in the document image to be detected specifically includes: Determine a confidence threshold corresponding to the document image to be detected according to a preset number; Determine the feature data of each anchor frame in the document image to be detected according to the confidence threshold; wherein the number of anchor frames in the document image to be detected does not exceed the preset number.

5. The method according to any one of claims 1 to 4, characterized in that: Before acquiring the document image to be detected, including: Acquire a training data set, wherein the training data set includes a plurality of training document images and coordinates of each text box corresponding to each training document image; The training model is iteratively trained using the training data set to obtain a trained image detection model; The training document image is obtained by performing image restoration processing on the document source image, the number of layout image categories corresponding to the training data set is not less than the preset number of categories, and the difference in the number of layout images of different categories does not exceed the preset difference.

6. An image processing device, characterized in that: include: An acquisition module, used for acquiring the image of the document to be detected; A processing module, used for processing the document image to be detected by means of a trained image detection model to obtain various layout data in the document image to be detected; the image detection model includes a query module, an OCR module, a fusion module and a local decoding module; the layout data includes a layout image and a corresponding category; Among them, the query module is used to determine the feature data of each anchor frame in the document image to be detected, the OCR module is used to determine the coordinates of each text frame of the document image to be detected, and the fusion module is used to generate the coordinates of each enhanced anchor frame of the document image to be detected according to the anchor frame feature data and text frame coordinates of the document image to be detected; the local decoding module is used to determine the local feature data of the enhanced anchor frame based on the local range corresponding to each enhanced anchor frame coordinate, and obtain the layout data corresponding to each enhanced anchor frame.

7. A business data system, characterized in that: include: Document analysis unit and layout recognition unit; The document analysis unit is used to perform the method according to any one of claims 1 to 5; The layout recognition unit is used to receive layout images of various categories output by the document analysis unit, and determine information in the layout image based on the category of the layout image.

8. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 5 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 5 when being executed by a processor.

Images