Multi-modal structured data intelligent identification method, device and system and storage medium

Abstract

The invention provides a multi-modal structured data intelligent identification method, device and system and a storage medium, and the method at least comprises the following steps: obtaining a multi-modal picture comprising a table image and a table text, and respectively extracting visual features and text features; according to position information and / or format information in the text features and wireframe information in the visual features, the visual features and the text features are fused, and multi-modal fusion features are obtained; and based on the multi-modal fusion features, decoding and outputting corresponding structured data containing the multi-modal picture. According to the multi-modal structured data intelligent identification method provided by the invention, the table image and the table text as well as the visual features and the text features corresponding to the table image and the table text are obtained by analyzing the multi-modal picture, the two features are analyzed respectively, and the identification accuracy of the multi-modal structured data is improved by means of the characteristics of the text features. And the computer is reused to fuse the text features and the visual features, and finally structured data which can be processed by the computer is decoded.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for intelligent recognition of multimodal structured data. Background Technology

[0002] OCR, or Optical Character Recognition, is a traditional technology that uses multimodal structured data intelligent recognition systems (such as scanners or digital cameras) to examine characters printed on paper. It determines the shape of characters by detecting dark and light patterns, and then uses character recognition methods to translate the shapes in the photograph into computer text. Although computer recognition technology has made significant progress and the recognition accuracy of OCR technology has continuously improved, it still has limitations in handling practical work situations.

[0003] In the field of worksheets, such as complex table structures like merged cells and nested tables, unimodal methods cannot effectively associate visual features with semantic information. Furthermore, existing table recognition systems have poor generalization capabilities, struggling to adapt to tables with different templates, resolutions, tilts, and / or display areas. Text detection and table structure recognition are often processed separately, leading to accumulated recognition errors and severely distorted results. Summary of the Invention

[0004] Based on this, in order to solve at least one of the problems mentioned above, the present invention provides a method, apparatus, system and storage medium for intelligent recognition of multimodal structured data.

[0005] In a first aspect, the present invention provides a method for intelligent recognition of multimodal structured data, comprising at least the following steps: A multimodal image containing both table images and table text was obtained, and visual features and text features were extracted from it. Based on the positional and / or format information in the text features and the wireframe information in the visual features, the visual features and the text features are fused to obtain multimodal fusion features; Based on the multimodal fusion features, the decoding output contains structured data corresponding to the multimodal image.

[0006] In some implementations of the first aspect, the location information includes first coordinate information of the table text on the multimodal image, and the wireframe information includes second coordinate information of visible lines in the table image; the step of fusing the visual features and the text features includes: Based on the first coordinate information and the second coordinate information, the data sequence corresponding to the text feature and the data sequence corresponding to the visual feature are resampled into an associated data sequence of the same length.

[0007] Combining the first aspect and the above implementation methods, in some implementation methods of the first aspect, the format information includes the length, width, and number of rows of the table text, and the wireframe information includes the second coordinate information of the visible lines in the table image; the step of fusing the visual features and the text features includes: Based on the format information and the second coordinate information, the data sequence corresponding to the text feature is standardized and transformed to obtain the fused text feature; Based on the location information and the wireframe information, the data sequence corresponding to the fused text feature and the data sequence corresponding to the visual feature are resampled into an associated data sequence of the same length.

[0008] In combination with the first aspect and the above implementation methods, in some implementation methods of the first aspect, the position information further includes the tilt angle of the table text; the step of standardizing and transforming the data sequence corresponding to the text feature further includes: adding deflection angle data to both the data sequence corresponding to the text feature and the data sequence corresponding to the visual feature according to the tilt angle.

[0009] Combining the first aspect and the above implementation methods, in some implementation methods of the first aspect, the steps of extracting visual features and text features respectively include: The OCR recognition engine is invoked to recognize the table text and obtain the text features, which include several independent text blocks and the third coordinate information of each text block.

[0010] Combining the first aspect and the above implementation methods, in some implementation methods of the first aspect, the wireframe information includes the second coordinate information of the visible lines in the table image; the step of fusing the visual features and the text features further includes: Based on the third coordinate information, the second coordinate information is corrected by a preset data difference to obtain the corrected second coordinate information; Based on the third coordinate information and the corrected second coordinate information, the data sequence corresponding to the text block and the data sequence corresponding to the visual feature are resampled into an associated data sequence of the same length.

[0011] In combination with the first aspect and the above implementation methods, in some implementation methods of the first aspect, the step of obtaining a multimodal image including a table image and table text includes scanning or photographing a document containing table content.

[0012] Secondly, the present invention provides a multimodal structured data intelligent recognition device, comprising: The acquisition module is used to acquire multimodal images including table images and table text, and extract visual features and text features respectively; The fusion module is used to fuse the visual features and the text features based on the position information and / or format information in the text features and the wireframe information in the visual features to obtain multimodal fusion features; The decoding module is used to decode and output structured data containing the multimodal image based on the multimodal fusion features.

[0013] Thirdly, the present invention provides a multimodal structured data intelligent recognition system, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the multimodal structured data intelligent recognition method as described in any one of the first aspects of the present invention.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal structured data intelligent recognition method described in any one of the first aspects of the present invention.

[0015] The technical solutions provided in the embodiments of the present invention bring the following beneficial technical effects: The multimodal structured data intelligent recognition method provided by this invention obtains table images and table text, as well as corresponding visual and text features, by parsing multimodal images. It analyzes the two types of features separately, and uses the inherent characteristics of text features to re-integrate the text features and visual features using a computer. Finally, it decodes the structured data that can be processed by the computer. Even if the table template is different, the table clarity is different, or even if the position is skewed, it can effectively generate structured data corresponding to the table image.

[0016] Additional aspects and advantages of this application will be set forth in the following sections and will be understood in detail from the following description, or may be learned by specific practice of the invention. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a multimodal structured data intelligent recognition method according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of fusing visual features and text features in one embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of achieving structured data output through a multimodal structured data intelligent recognition method in one embodiment of the present invention; Figure 4 This is a schematic diagram of the logic flow of the multimodal structured data intelligent recognition method for fusion processing in one embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the process of reconstructing a table image using a multimodal structured data intelligent recognition method according to an embodiment of the present invention. Figure 6 This is a schematic diagram of the structural framework of a multimodal recognition table device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structural framework of a multimodal structured data intelligent recognition system according to an embodiment of the present invention. Detailed Implementation

[0019] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0020] It should be noted that when a component is said to be "fixed to" another component, it can be directly attached to the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] An embodiment of the first aspect of the present invention provides a method for intelligent recognition of multimodal structured data, such as... Figure 1 As shown, it includes at least the following steps: S100: Obtain a multimodal image including a table image and table text, and extract visual features and text features respectively. In S100, the multimodal image is mainly interpreted and analyzed from two aspects: the table outline and the text set, converting them into visual features corresponding to the table image and text features corresponding to the table text. As is already achievable in existing technology, computers can read text and lines using OCR technology, converting them into internal computer data, recorded as text features and visual features respectively. Specifically, the steps for obtaining a multimodal image including a table image and table text include scanning or photographing the document containing the table content. The application scenario of this invention is for how to identify tables recorded on paper in real-life work and directly and accurately convert them into editable tables in a computer. Therefore, the method of obtaining multimodal images is usually to record the table on paper into a computer device or a multimodal structured data intelligent recognition system by photographing or scanning, and then perform subsequent processing using conventional OCR technology.

[0023] S200: Based on the positional and / or formatal information in the text features and the wireframe information in the visual features, the visual features and text features are fused to obtain multimodal fused features. Text features include positional and formatal information. Positional information refers to the specific location of the text in the multimodal image, while formatal information includes the length, width, and number of lines of the area occupied by the text. Fusing visual and text features means aligning text features with visual features. This alignment is a soft alignment concept within the model, rather than a strict one-to-one correspondence. The criterion is that after processing by the fusion encoder, for each token in the text sequence, the model can find sufficient visual evidence supporting its semantics and location in its corresponding resampled visual features. During training, this alignment is driven by a loss function. For example, when predicting the coordinates of a cell, the model simultaneously uses the text token within that cell and its corresponding visual features, i.e., the area outlined by the wireframe. If the alignment is successful, the model can accurately associate a text with its specific location in the image. After the model training is complete, the attention weight map can be visualized to see whether a text token (e.g., "amount") falls into the visual region corresponding to the table in the image. Through S200, the visual features and text features that were originally read separately are fused or "aligned" to form multimodal fused features.

[0024] S300: Based on multimodal fusion features, it decodes and outputs structured data corresponding to multimodal images. The computer can decode and output structured data, such as HTML, JSON, and CSV data, based on multimodal fusion features. The computer can then read this structured data and create editable tables.

[0025] The multimodal structured data intelligent recognition method provided by this invention obtains table images and table text, as well as corresponding visual and text features, by parsing multimodal images. It analyzes the two types of features separately, and uses the inherent characteristics of text features to re-integrate the text features and visual features using a computer. Finally, it decodes the structured data that can be processed by the computer. Even if the table template is different, the table clarity is different, or even if the position is skewed, it can effectively generate structured data corresponding to the table image.

[0026] Specifically, in some implementations of the first aspect of the present invention, the location information includes the first coordinate information of the table text on the multimodal image, and the wireframe information includes the second coordinate information of the visible lines in the table image; the step of fusing visual features and text features includes: resampling the data sequence corresponding to the text features and the data sequence corresponding to the visual features into an associated data sequence of the same length according to the first coordinate information and the second coordinate information. By making full use of location information and size information, the read image information can be processed by a computer according to specific settings to obtain structured data that is more consistent with the multimodal image. Table text, that is, the text in the table, usually includes several characters. If it is Chinese characters, it often has at least one Chinese character. This text will have a certain position on the image, and this position can be determined by the computer at the same time after the text features are read. The table is composed of vertical, horizontal or diagonal lines, especially the visible lines. After the computer reads the visual features corresponding to the table image, it will inevitably form the line number of these lines, the length value of each line, the start coordinate, the end coordinate, etc.

[0027] To fuse visual and textual features, a deep semantic association needs to be built between visual features (including information such as lines, cell layout, and cell position) and textual features (including text, numbers, symbols, and their size and position). The first step is spatial alignment. Using the text bounding box coordinates provided by OCR, each text token is bound to its specific spatial location in the image (i.e., the corresponding visual region). Attention is paid to the visual context surrounding a given text, such as the cell's border thickness, shadow, and color, as well as the text within a visual region. The surrounding cells are mapped to each other, and the two are merged. In the computer, their respective data sequences are resampled and fused. This is specifically achieved through a perceptual resampler, for example, by converting the data sequence corresponding to the visual features V = [v1, v2, ..., v...]. k Resampling is performed on a data sequence T = [t1, t2, ..., t] corresponding to the text features. m A data sequence of uniform length V' = [v'1, v'2, ..., v' m ].

[0028] Secondly, semantic alignment is performed using a cross-modal attention mechanism, allowing visual and textual features to map to each other and achieve a mutual query effect. For example, textual features can act as a "query" to "retrieve" (attention) the most relevant visual features; conversely, visual features (such as a pattern of merged cells) can also act as a "query" to "find" the key text describing it. The data sequence T corresponding to the text features is used as the query, and the sequence V corresponding to the visual features is used as the key and value. An attention calculation is performed: Attention(Q=T, K=V, V=V), abbreviated as weight matrix A, and then alignment weights are generated. The weight matrix A (of size m x k) output by the above attention represents the degree of attention each text token gives to each image region. The visual features V are weighted and summed using the weight matrix A to obtain a preliminary fusion feature. Then, a learnable feedforward neural network is used to map this preliminary fusion feature to a space of the same dimension as the text features, finally outputting the resampled visual feature V'. After this step, each visual feature v' in V'... i All are related to text features t i The most relevant visual context information is condensed. Furthermore, the resampled visual features V' are deeply fused with the text features T: a multimodal Transformer layer is used to concatenate V' and T into a long multimodal sequence X = [v'1, v'2, ..., v']. m, t1, t2, ..., t m The multimodal sequence X is input into a standard Transformer encoder layer. Within this layer, a self-attention mechanism ensures that each token (whether visual or textual) interacts with all other tokens in the sequence. For example, a text token t... j It will simultaneously monitor the text tokens surrounding it, as well as the visual token v' aligned with it. j It may also pay attention to visual tokens in other rows and columns. Each token undergoes self-attention and then a non-linear transformation through a feedforward network. Such multimodal Transformer layers are stacked N layers (e.g., 6 layers) to achieve deep cross-modal understanding.

[0029] Specifically, in conjunction with the embodiments of the first aspect and the above implementation methods, in some other embodiments of the first aspect, the format information includes the length, width, and number of rows of the table text, and the wireframe information includes the second coordinate information of the visible lines in the table image; the step of fusing visual features and text features includes: standardizing the data sequence corresponding to the text features according to the format information and the second coordinate information to obtain fused text features; and then resampling the data sequence corresponding to the fused text features and the data sequence corresponding to the visual features into an associated data sequence of the same length according to the position information and the wireframe information.

[0030] As described earlier, tabular text, or text within a table, typically consists of several characters. These characters exhibit a certain regular format, including the spacing between characters, the length of each character, and the width of the string—this constitutes the format information. Through OCR technology, this format information related to text features is transformed into data that can be processed by a computer. In this embodiment, based on the alignment of text within a cell with text in other areas, such as the top, bottom, left, or right areas—that is, the format information—it is "inferred" that this area of ​​the cell should have a cell boundary. Formally, the text boundary corresponding to the format information extends outward by a specific size. This specific size can be considered an editable parameter that can be manually set and adjusted. Processing is done uniformly according to this editing parameter, and a standardized transformation is performed to form a new text with a new text boundary—the aforementioned fused text feature. This new text is then fused with visual features to form a related data sequence. Through the above processing in this embodiment, even if this "cell boundary" does not exist in the visual features—that is, if there are implicit borders in the table—the text can still be correctly "filled" into the table, ultimately forming structured data.

[0031] In conjunction with the embodiments of the first aspect and the above implementation methods, in some other implementation methods of the embodiments of the first aspect, the position information also includes the tilt angle of the table text; the step of standardizing and transforming the data sequence corresponding to the text features further includes: adding deflection angle data to both the data sequence corresponding to the text features and the data sequence corresponding to the visual features according to the tilt angle. In the prior art, Hough transform or contour detection is usually used to find straight lines, and then the tilt angle is calculated for rotation. In this invention, the tilt angle of the text in the table is first recorded. This tilt angle can be determined based on the statistical data of the tilt angles of multiple sets of text in the table. The text is tilted, and even if the surrounding tables or cells are not explicit tables, the tables that can be displayed can still be processed. Specifically, the minimum bounding rectangle of the text lines is obtained using an OCR engine, the tilt angles of all text lines are counted, and the median is taken as the tilt angle of the text direction. This can effectively correct the text deformation caused by the shooting angle or the paper placement posture. At the same time, the outermost border line of the table can also be found through line detection to evaluate the border tilt angle. If the border line has high confidence and is continuous, its direction is used for secondary verification. The final correction angle = α * text tilt angle + β * border tilt angle. In this way, even if the table borders are incomplete, we can achieve accurate correction using a large amount of text information.

[0032] Furthermore, in some specific implementations, task-feedback-based adaptive enhancement is employed. This involves a rapid "pre-identification" of the original multimodal image to diagnose its main problems, such as low contrast, background noise, uneven lighting, and moiré patterns. For low contrast / uneven lighting, the CLAHE algorithm is used, with its parameters (grid size, contrast limit) adaptively adjusted according to image resolution. For background noise, non-local mean denoising is used instead of Gaussian filtering to better preserve line and character edges. For moiré patterns, specific frequency domain filtering methods are used to eliminate them. These correction processes aim to create a better input foundation for subsequent line detection and OCR recognition. Moreover, a detection model fine-tuned on a large number of table images is used. Its backbone neural network is more sensitive to line and text features, filtering out interference from non-table lines such as headers, footers, stamps, and handwritten annotations. In reality, many tables suffer from problems such as skewed scanning, dotted lines, broken lines, blemishes, and lines that are too thick or too thin. Purely visual methods are prone to failure. The solution provided by this invention relies on the lines when they are clear, and on the text layout and semantics as a fallback when the lines are blurry, thus ensuring the recognition rate.

[0033] Optionally, in conjunction with the foregoing embodiments and implementations, in some further embodiments of the first aspect, the steps of extracting visual features and text features respectively include: calling an OCR recognition engine to recognize the table text and obtaining text features, wherein the text features include several independent text blocks and the third coordinate information of each text block. In many tables, each cell may contain more than one line, but according to semantic recognition technology, a computer can interpret text with more than one line as a statement expressing a meaning, that is, to regard multiple lines of text as a text block, one semantic text corresponds to one text block, and each text block corresponds to one coordinate information, namely, the third coordinate information. Furthermore, in a specific implementation of the present invention, such as Figure 2 As shown, the wireframe information includes the second coordinate information of the visible lines in the table image; the step of fusing visual features and text features in S200 also includes: S210: Based on the third coordinate information, the second coordinate information is corrected using a preset data difference to obtain the corrected second coordinate information.

[0034] S220: Based on the third coordinate information and the corrected second coordinate information, resample the data sequence corresponding to the text block and the data sequence corresponding to the visual features into an associated data sequence of the same length.

[0035] This implementation first determines whether the table is a "fully wired table," a "semi-wired table," or a "wireless table," providing the foundation for subsequent processing. Secondly, when the table is a semi-wired or wireless table, such as a common three-line table, the text blocks within the table indicate that cells actually exist around them, though they are not explicitly displayed. Using the method described in the previous embodiments, it can be "inferred" that the area containing the text block should have a cell boundary. Formally, this means extending the third coordinate information corresponding to the text block outwards by a specific size. This specific size can be considered an editable parameter that can be manually set or adjusted. The data is then processed according to this editable parameter, undergoing standardized transformation to form new second coordinate information. In other words, the implicit lines in the original semi-wired or wireless table are presented as data, resulting in corrected second coordinate information. This redrawn table is then merged with the text to form a related data sequence. "Explicit line detection" provides the "physical structure seen by the eye," while the analysis and processing of text blocks provides "implicit structure perception," offering a "logical structure that the brain can understand but does not yet possess a physical structure." The method provided by this invention possesses both of these capabilities and can intelligently integrate them, enabling it to understand ever-changing tables more flexibly and accurately, much like a human.

[0036] Regarding the ability to generalize to structural diversity, the solution provided by this invention uses a unified model that can handle "fully wired tables," "fully wireless tables," and "semi-wired tables" that fall between the two. The aforementioned "implicit structure awareness" steps can accurately infer the range of merged cells by analyzing text blocks spanning rows and columns, alignment violations, etc., without needing to train or switch different models for different table styles, significantly improving practical value.

[0037] In some practical cases, the above methods and existing technologies can be combined to achieve the output of structured data. For ease of understanding, the detailed process is listed below: like Figure 3 As shown, the input table image undergoes preprocessing, primarily in three aspects: image correction, image enhancement, and object detection. Image correction includes image distortion removal and rotation, while image enhancement mainly involves adjusting contrast and binarization. Object detection primarily involves detecting table regions. After obtaining the table image, it is processed by the text encoder and visual encoder of the multimodal large model core to obtain visual and text features. These features are then passed through a fusion encoder, perceptual resampling, and attention fusion, and processed by the large language model core, such as existing LLaMA, GPT, and ChatGLM models. After post-processing and output, structured data in JSON, CSV, or Excel format is obtained.

[0038] The above process, refer to Figure 4 This allows for a deeper understanding of the content. The input layer already contains the original high-resolution image and may also include the user's query text. After being segmented into image blocks and / or tokenized, the original visual feature sequence and text feature sequence are obtained through a visual encoder and a text encoder, respectively. The number of tokens in the two feature sequences may not be the same, but after processing by a perceptual resampling unit, an aligned visual token sequence is output. The core of input fusion and understanding combines the information carried by the text feature sequence as an instruction prefix, which is then processed by a large language model into an understood fusion representation, generating an answer that provides integrated image and text information.

[0039] refer to Figure 5 In terms of technical methodology, the process begins by acquiring projection data, which is then subjected to Fourier transform to convert each projection line to the frequency domain. After frequency domain gridding, the fan-beam projection data is interpolated to a Cartesian rectangular coordinate grid. An inverse Fourier transform is then performed to convert the gridded frequency domain data back to image space, resulting in the reconstructed image in the computer. The image quality is then assessed to determine if it meets the requirements. If it does, the reconstructed image is output; otherwise, parameters are adjusted, filtering functions and interpolation methods are used, and the process returns to the Fourier transform step for further processing until the final reconstructed image is output.

[0040] To facilitate a better understanding of the multimodal structured data intelligent recognition method provided by this invention, a practical application example is presented in tabular form below. This example processes a slightly tilted invoice image with merged cells into structured data that can be processed by a computer and displayed on a computer screen. See Table 1 below:

[0041] A second aspect of the present invention provides a multimodal structured data intelligent recognition device 10, such as... Figure 6 As shown, it includes an acquisition module 11, a fusion module 12, and a decoding module 13. Wherein: The acquisition module 11 is used to acquire multimodal images including table images and table text, and extract visual features and text features respectively.

[0042] The fusion module 12 is used to fuse visual features and text features based on the positional information and / or format information in the text features and the wireframe information in the visual features to obtain multimodal fused features.

[0043] The decoding module 13 is used to decode and output structured data corresponding to the multimodal images based on multimodal fusion features.

[0044] Specifically, the location information includes the first coordinate information of the table text on the multimodal image, and the wireframe information includes the second coordinate information of the visible lines in the table image; the fusion module 12 fuses the visual features and text features, including: based on the first coordinate information and the second coordinate information, resampling the data sequence corresponding to the text features and the data sequence corresponding to the visual features into an associated data sequence of the same length.

[0045] Specifically, the format information includes the length, width, and number of rows of the table text, and the wireframe information includes the second coordinate information of the visible lines in the table image; the fusion module 12 integrates the visual features and text features in the following steps: Based on the format information and the second coordinate information, the data sequence corresponding to the text features is standardized and transformed to obtain the fused text features; Based on location and wireframe information, the data sequences corresponding to the fused text features and the data sequences corresponding to the visual features are resampled into associated data sequences of the same length.

[0046] Furthermore, the location information also includes the tilt angle of the table text; the step of standardizing the data sequence corresponding to the text features also includes: adding the tilt angle data to both the data sequence corresponding to the text features and the data sequence corresponding to the visual features, based on the tilt angle.

[0047] Specifically, the steps of obtaining visual features and text features by module 11 include: calling the OCR recognition engine to recognize the table text and obtaining text features, which include several independent text blocks and the third coordinate information of each text block.

[0048] Furthermore, the wireframe information includes the second coordinate information of the visible lines in the table image; the step of the acquisition module 11 in fusing visual features and text features also includes: correcting the second coordinate information based on the third coordinate information with a preset data difference to obtain corrected second coordinate information; and resampling the data sequence corresponding to the text block and the data sequence corresponding to the visual features into an associated data sequence of the same length based on the third coordinate information and the corrected second coordinate information.

[0049] Optionally, the step of acquiring the multimodal image including table image and table text by the acquisition module 11 includes scanning or photographing the document containing the table content.

[0050] Based on the same inventive concept, and referring to Figure 7 The third aspect of the present invention provides a multimodal structured data intelligent recognition system 1000, including: a processor 1001, a memory 1003 and a computer program stored in the memory 1003, wherein the processor 1001 and the memory 1003 are electrically connected, such as being connected through a bus 1002, and the processor 1001 executes the computer program to implement the multimodal structured data intelligent recognition method of any of the first aspects of the present invention.

[0051] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 1001 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0052] Bus 1002 may include a pathway for transmitting information between the aforementioned components. Bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0053] The memory 1003 may be ROM (Read-Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read-Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0054] Those skilled in the art will understand that the multimodal structured data intelligent recognition system 1000 provided in this embodiment of the invention can be specifically designed and manufactured for the desired purpose, or it may include known devices in general-purpose computers. These devices have computer programs stored therein that are selectively activated or reconfigured. Such computer programs can be stored in a device (e.g., computer) readable medium or in any type of medium suitable for storing electronic instructions and respectively coupled to a bus.

[0055] The multimodal structured data intelligent recognition system 1000 provided by this invention runs a computer program that records a multimodal structured data intelligent recognition method. By parsing multimodal images, it obtains table images and table text, as well as corresponding visual and text features. It analyzes the two types of features separately, and by leveraging the inherent characteristics of text features, it reuses the computer to fuse the text features and visual features. Finally, it decodes the structured data that the computer can process. Even if the table template is different, the table clarity is different, or even if the position is skewed, it can effectively generate structured data corresponding to the table image.

[0056] Specifically, the multimodal structured data intelligent recognition system 1000 includes a transceiver 1004. The transceiver 1004 can be used for signal reception and transmission. The transceiver 1004 allows the multimodal structured data intelligent recognition system 1000 to communicate wirelessly or wiredly with other devices to exchange data. It should be noted that in practical applications, the transceiver 1004 is not limited to one unit.

[0057] Specifically, the multimodal structured data intelligent recognition system 1000 includes an input unit 1005. The input unit 1005 can be used to receive input numbers, characters, and / or images, or generate key signal inputs related to user settings and function control of the multimodal structured data intelligent recognition system 1000. The input unit 1005 may be one or more of the following: a touch screen, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, a joystick, a camera, a scanner, etc.

[0058] Specifically, the multimodal structured data intelligent recognition system 1000 also includes an output unit 1006. The output unit 1006 can be used to output or display information processed by the processor 1001. The output unit 1006 may include, but is not limited to, one or more of a display device, a speaker, a vibration device, etc.

[0059] Although Figure 7 A multimodal structured data intelligent recognition system 1000 with various devices is shown; however, it should be understood that implementation or possession of all the devices shown is not required. More or fewer devices may be implemented alternatively.

[0060] Optionally, the memory 1003 is used to store application code for executing the present invention, and its execution is controlled by the processor 1001. The processor 1001 is used to execute the application code stored in the memory 1003 to implement any of the multimodal structured data intelligent recognition methods provided in the embodiments of the present invention.

[0061] Based on the same technical concept, an embodiment of the fourth aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal structured data intelligent recognition method of any one of the first aspects of the present invention.

[0062] Because it contains a computer program with the intelligent recognition method for multimodal structured data provided by this invention as its logical core, the computer-readable storage medium provided by this invention can be used in various computer devices. It can obtain table images and table text, as well as corresponding visual features and text features, by parsing multimodal images. It can analyze the two types of features separately, and reuse the characteristics of the text features to fuse the text features and visual features together. Finally, it can decode the structured data that the computer can process. Even if the table template is different, the table clarity is different, or even if the position is skewed, it can effectively generate structured data corresponding to the table image.

[0063] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for intelligent recognition of multimodal structured data, characterized in that, It should include at least the following steps: A multimodal image containing both table images and table text was obtained, and visual features and text features were extracted from it. Based on the positional and / or format information in the text features and the wireframe information in the visual features, the visual features and the text features are fused to obtain multimodal fusion features; Based on the multimodal fusion features, the decoding output contains structured data corresponding to the multimodal image.

2. The multimodal structured data intelligent recognition method according to claim 1, characterized in that, The location information includes the first coordinate information of the table text on the multimodal image, and the wireframe information includes the second coordinate information of the visible lines in the table image; the step of fusing the visual features and the text features includes: Based on the first coordinate information and the second coordinate information, the data sequence corresponding to the text feature and the data sequence corresponding to the visual feature are resampled into an associated data sequence of the same length.

3. The multimodal structured data intelligent recognition method according to claim 1, characterized in that, The format information includes the length, width, and number of rows of the table text; the wireframe information includes the second coordinate information of the visible lines in the table image; the step of fusing the visual features and the text features includes: Based on the format information and the second coordinate information, the data sequence corresponding to the text feature is standardized and transformed to obtain the fused text feature; Based on the location information and the wireframe information, the data sequence corresponding to the fused text feature and the data sequence corresponding to the visual feature are resampled into an associated data sequence of the same length.

4. The multimodal structured data intelligent recognition method according to claim 3, characterized in that, The location information also includes the tilt angle of the table text; the step of standardizing and transforming the data sequence corresponding to the text features further includes: adding deflection angle data to both the data sequence corresponding to the text features and the data sequence corresponding to the visual features according to the tilt angle.

5. The intelligent recognition method for multimodal structured data according to claim 1, characterized in that, The steps of extracting visual features and text features respectively include: The OCR recognition engine is invoked to recognize the table text and obtain the text features, which include several independent text blocks and the third coordinate information of each text block.

6. The multimodal structured data intelligent recognition method according to claim 5, characterized in that, The wireframe information includes the second coordinate information of the visible lines in the table image; the step of fusing the visual features and the text features further includes: Based on the third coordinate information, the second coordinate information is corrected by a preset data difference to obtain the corrected second coordinate information; Based on the third coordinate information and the corrected second coordinate information, the data sequence corresponding to the text block and the data sequence corresponding to the visual feature are resampled into an associated data sequence of the same length.

7. The intelligent recognition method for multimodal structured data according to claim 1, characterized in that, The step of obtaining a multimodal image including table images and table text includes scanning or photographing a document containing table content.

8. A multimodal structured data intelligent recognition device, characterized in that, include: The acquisition module is used to acquire multimodal images including table images and table text, and extract visual features and text features respectively; The fusion module is used to fuse the visual features and the text features based on the position information and / or format information in the text features and the wireframe information in the visual features to obtain multimodal fusion features; The decoding module is used to decode and output structured data containing the multimodal image based on the multimodal fusion features.

9. A multimodal structured data intelligent recognition system, comprising: A memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the multimodal structured data intelligent recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal structured data intelligent recognition method according to any one of claims 1 to 7.

Images