Multimodal large model sales assistant product document structure recognition method and system
Through the visual-text feature alignment module of the multimodal large model, end-to-end document structure recognition is realized, solving the problem that single-modal models in the prior art cannot use text and visual information at the same time, improving the recognition effect and reducing dependence on labeled data.
Patent Information
- Application Number
- CN202411528841.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-04-04
AI Technical Summary
In the prior art, single-modal models cannot use text and visual information at the same time, resulting in limited document structure recognition effect, high system complexity, and large dependence on the scale of labeled data.
Using a multimodal large model, document structure recognition is performed end-to-end through visual feature extractor, text feature extractor and visual-text feature alignment module to generate content block location, type and reading order.
It realizes the use of text and visual information to identify the document structure simultaneously, reduces the dependence on the amount of labeled data, improves the recognition effect, and has low migration cost for new scenarios.
Smart Images
Figure CN119763127A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of multimodal large models, and in particular to a method and system for identifying the structure of a sales assistant product document of a multimodal large model. Background Art
[0002] Document structure recognition is a long-standing technical scenario. Both rule-based solutions and solutions based on traditional text and visual models have inherent defects, resulting in limited recognition effects. Moreover, the understanding of product manual documents directly affects the upper limit of the effectiveness of intelligent sales assistants, which is very important. With the development of technology in recent years, many new cutting-edge technologies have been provided for the optimization of this scenario.
[0003] Unlike traditional models that can only process one type of input, with the advancement of deep learning, multimodal models that can process multiple types of input at the same time and are more in line with the real physical world have developed rapidly in recent years, enabling them to perform well in many tasks, including natural language processing, computer vision, and automatic speech recognition.
[0004] Furthermore, thanks to the outstanding results of GPT and other works, generative models have received a lot of attention in recent years. Thanks to the flexibility of generative training objectives, it can be migrated to almost all scenarios for end-to-end training, so in recent years, large multimodal models based on generative models have received more and more attention.
[0005] The most famous structure recognition model in the existing technical solutions comes from LayoutLMv3 proposed by Microsoft. To achieve complete document structure recognition, it mainly includes three main steps:
[0006] Step 1: Pre-train a basic feature extraction model based on the Transformer architecture using a large amount of graphic data
[0007] Step 2: Extract visual features of the document image through the model obtained in step 1, and train an object detection model that can identify content block areas based on the features;
[0008] Step 3: Based on the prediction results given by the model in step 2, the sequence generation model is trained to obtain the reading order of the content blocks.
[0009] Overall, the model training involves three times, and the models in step 2 and step 3 need to be deployed as services before they can be used in real scenarios.
[0010] That is, in the existing technology, on the one hand, the unimodal model cannot utilize text and visual information at the same time, while humans read documents by considering both text and visual information;
[0011] Secondly, traditional models cannot provide an end-to-end solution. They need to first identify the content area, then the content type, and finally restore the reading order of the content (three models are required to solve the problem), which results in high system complexity.
[0012] Third, traditional models rely heavily on the scale of labeled data, requiring tens of thousands of labeled data.
[0013] Finally, the next word prediction loss of the generative model cannot be aligned with the true loss of the target box area. Summary of the invention
[0014] In view of this, an object of the present invention is to provide a method and system for identifying the structure of a sales assistant product document of a multimodal large model to solve the problems existing in the prior art.
[0015] According to a first aspect of an embodiment of the present invention, a method for identifying a sales assistant product document structure of a multimodal large model is provided, comprising:
[0016] Obtain product document information of the sales assistant to be identified;
[0017] The sales assistant product document information to be identified is obtained, input into a preset multimodal model, and the identified sales assistant product document information is output.
[0018] Furthermore, the pre-set multimodal model includes: Qwen-VL 2 multimodal model.
[0019] Furthermore, the method of obtaining the sales assistant product document information to be identified, inputting the preset multimodal model, and outputting the identified sales assistant product document information includes:
[0020] The visual feature information in the sales assistant product document information to be identified is extracted using a preset visual feature extractor; wherein the visual feature includes: coordinate information;
[0021] The visual feature information in the sales assistant product document information to be identified is extracted using a preset text feature extractor; wherein the text feature includes: tags in HTML language;
[0022] The visual feature information in the sales assistant product document information to be identified is aligned with the visual feature information in the sales assistant product document information to be identified using a preset visual-text feature alignment module, and the identified sales assistant product document information is output.
[0023] Furthermore, the Qwen-VL 2 multimodal model comprises:
[0024] Visual feature extractor, text feature extractor, visual-text feature alignment module.
[0025] Furthermore, the visual encoder adopts a Vision Transformer architecture to process an input image and generate a set of image features.
[0026] Furthermore, the visual-text feature alignment module is used to compress the feature sequence of the visual feature information in the sales assistant product document information to be identified extracted by the visual feature extractor, and to align the visual feature information in the sales assistant product document information to be identified with the visual feature information in the sales assistant product document information to be identified, and output the identified sales assistant product document information.
[0027] Furthermore, the Qwen-VL 2 multimodal model further includes:
[0028] Regression loss function, the expression of the regression loss function is as follows:
[0029]
[0030] Among them, Area of Overlap: represents the intersection between the detected text box and the standard text box; Area of Union: represents the union between the detected text box and the standard text box.
[0031] According to a second aspect of an embodiment of the present invention, a sales assistant product document structure recognition system of a multimodal large model is provided, which is applied to any of the above-mentioned sales assistant product document structure recognition methods of a multimodal large model, and the system includes:
[0032] An acquisition module, used to acquire product document information of a sales assistant to be identified;
[0033] The processing module is used to obtain the sales assistant product document information to be identified, input it into a preset multimodal model, and output the identified sales assistant product document information.
[0034] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:
[0035] It is understandable that the technical solution provided by the present invention obtains the sales assistant product document information to be identified; the sales assistant product document information to be identified is input into a pre-set multimodal model, and the identified sales assistant product document information is output. It is understandable that the technical solution provided by the present invention utilizes a large multimodal model to simultaneously utilize text and visual information, and simultaneously utilizes text and visual information to perform document structure recognition, with excellent recognition effect; in addition, it can reduce the dependence on the amount of annotated data, and at the same time, the cost of migrating to new scenes is low; an end-to-end model solution can be generated, and the content block position, type, and reading order can be generated at the same time, and the limitations of the generative loss can be compensated by proposing a multi-task loss that combines the generative loss and the region box regression loss.
[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0038] Figure 1 It is a schematic diagram of steps of a method for identifying a sales assistant product document structure of a multimodal large model according to an exemplary embodiment;
[0039] Figure 2 It is a data conversion diagram of a method for identifying a sales assistant product document structure of a multimodal large model according to an exemplary embodiment;
[0040] Figure 3 It is a schematic diagram showing the composition of a multimodal large-model sales assistant product document structure recognition system according to an exemplary embodiment. DETAILED DESCRIPTION
[0041] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.
[0042] Embodiment 1
[0043] See also Figure 1 , Figure 1 The figure is a schematic diagram of steps of a method for identifying a sales assistant product document structure of a multimodal large model according to an exemplary embodiment, the method comprising:
[0044] S1. Obtain the product document information of the sales assistant to be identified;
[0045] S2. Obtain the sales assistant product document information to be identified, input it into a pre-set multimodal model, and output the identified sales assistant product document information.
[0046] In one embodiment, in a smart sales scenario, the core problem to be solved is how to find the corresponding content in the product manual based on the user's input requirements to provide reasonable product sales suggestions.
[0047] Compared to searching word by word, a smarter approach is to understand the structure of the document and search hierarchically:
[0048] -First locate the specific document by title;
[0049] -Then locate the specific chapter through the subheadings;
[0050] -Finally locate the specific content block (including: paragraphs, tables, pictures, etc.);
[0051] Since the formats of various product manuals vary, especially industrial products that contain a large number of CAD drawings, product parameter tables, flow charts and other field-specific content, it is necessary to be able to locate specific areas to facilitate the subsequent use of separate domain models for refined solutions.
[0052] Existing unimodal models cannot utilize text and visual information at the same time, while humans consider text and visual information simultaneously when reading documents. Traditional models cannot provide an end-to-end solution and need to first identify the content area, then identify the content type, and finally restore the reading order of the content (requiring three models to solve). The system complexity is high; traditional models rely heavily on the scale of annotated data and require tens of thousands of annotated data; the next word prediction loss of the generative model cannot be aligned with the actual loss of the target box area.
[0053] In order to solve the above problems, the present application obtains the sales assistant product document information to be identified; the sales assistant product document information to be identified is input into a pre-set multimodal model, and the identified sales assistant product document information is output. It can be understood that the technical solution provided by the present invention utilizes a large multimodal model to simultaneously utilize text and visual information, and simultaneously utilizes text and visual information to perform document structure recognition, with excellent recognition effect; in addition, it can reduce the dependence on the amount of annotated data, and the cost of migrating new scenes is low; an end-to-end model solution can be generated, and the content block position, type, and reading order can be generated at the same time, and the limitations of the generative loss can be compensated by proposing a multi-task loss that combines the generative loss and the region box regression loss.
[0054] In one embodiment, as described in step S2, the step of obtaining the sales assistant product document information to be identified, inputting the preset multimodal model, and outputting the identified sales assistant product document information includes: extracting visual feature information from the sales assistant product document information to be identified using a preset visual feature extractor; wherein the visual feature includes: coordinate information;
[0055] The visual feature information in the sales assistant product document information to be identified is extracted using a preset text feature extractor; wherein the text feature includes: tags in HTML language;
[0056] The visual feature information in the sales assistant product document information to be identified is aligned with the visual feature information in the sales assistant product document information to be identified using a preset visual-text feature alignment module, and the identified sales assistant product document information is output.
[0057] In the specific implementation, the annotation of generated data:
[0058] Since the multimodal large model adopts a generative model training scheme, we need a new annotation format that is different from the traditional scheme and supports the location, type, and order annotation of content blocks.
[0059] It should be noted that we used the LabelStudio open source labeling tool and selected the "target detection task" for labeling.
[0060] Specifically: provide a picture to be annotated, frame a rectangular area, which will correspond to four coordinate values (x0, y0) and (x1, y1);
[0061] The annotation area type will correspond to Header, Title and other types:
[0062] Mark the reading order, corresponding to serial numbers such as 1, 2, 3.
[0063] See also Figure 2 , the labeled data based on target detection will be converted into the multimodal model:
[0064] <ref> Header< / ref> <box> (666,0),(950,50)< / box>
[0065] <ref> Header< / ref> <box> (3,18),(60,150)< / box>
[0066] <ref> Header< / ref> <box> (7,162),(60,260)< / box>
[0067] <ref> Header< / ref> <box> (5,456),(70,590)< / box>
[0068] <ref> Title< / ref> <box> (818,81),(950,100)< / box>
[0069] <ref> Title< / ref> <box> (82,107),(210,130)< / box>
[0070] <ref> Title< / ref> <box> (82,146),(650,230)< / box>
[0071] <ref> Text< / ref> <box> (84,243),(660,340)< / box>
[0072] <ref> Title< / ref> <box> (84,346),(140,360)< / box>
[0073] <ref> Text< / ref> <box> (85,366),(650,490)< / box>
[0074] <ref> Text< / ref> <box> (85,499),(650,630)< / box>
[0075] <ref> Text< / ref> <box> (85,633),(650,760)< / box>
[0076] <ref> Text< / ref> <box> (83,766),(650,800)< / box>
[0077] <ref> Title< / ref> <box> (758,109),(850,130)< / box>
[0078] <ref> Text< / ref> <box> (674,132),(940,380)< / box>
[0079] <ref> Title< / ref> <box> (751,413),(860,440)< / box>
[0080] <ref> Text< / ref> <box> (676,439),(960,630)< / box>
[0081] <ref> Footer< / ref> <box> (132,937),(400,970)< / box>
[0082] in:
[0083] <ref>< / ref> The content in indicates the type of this area, header means header, footer means footer, title means title, and text means paragraph;
[0084] <box>< / box> The coordinates in represent the location of this area, and the four values represent:
[0085] The first value represents the distance from the upper left corner of the rectangular area to the left border;
[0086] The second value indicates the distance from the upper left corner of the rectangular area to the upper border;
[0087] The third value represents the distance from the right border to the lower right corner of the rectangular area.
[0088] The fourth value represents the distance from the lower right corner of the rectangular area to the lower border;
[0089] The order of the regions corresponds to the reading order.
[0090] Through this annotation method, we take into account three types of annotation information at the same time.
[0091] In one embodiment, the multimodal model selected in the embodiments of the present application includes: Qwen-VL 2 multimodal model.
[0092] Furthermore, the Qwen-VL 2 multimodal model comprises:
[0093] Visual feature extractor, text feature extractor, visual-text feature alignment module.
[0094] Furthermore, the visual encoder adopts a Vision Transformer architecture to process an input image and generate a set of image features.
[0095] Furthermore, the visual-text feature alignment module is used to compress the feature sequence of the visual feature information in the sales assistant product document information to be identified extracted by the visual feature extractor, and to align the visual feature information in the sales assistant product document information to be identified with the visual feature information in the sales assistant product document information to be identified, and output the identified sales assistant product document information.
[0096] In the specific implementation, the images in the sales assistant product document information to be identified will be converted into semantic tokens based on the vision encoder network, and then processed together with the prompt sentence through the language model text encoding network.
[0097] In the specific implementation, the tokens include two categories, one is the visual token extracted based on the visual information of the image area, and the other is the language token extracted based on the text characters. Since the tokens will eventually be mapped to vectors of the same dimension, it can be considered that after the model training, the alignment of the two is achieved.
[0098] In the specific implementation, the specific content of the prompt is not the focus of the protection of this application. As long as the same prompt is used in training and prediction, the effect can be achieved. Specifically, the prompt used in this application is: "Frame all content areas in the picture and return in reading order."
[0099] More specifically, the model uses a learnable network to compress the visual tokens output by VIT into a set of vectors with a fixed length of 256 and the same dimension as the LLM input. Through model training, these two embedding features can be distributed in the same semantic space, thereby achieving feature alignment between images and texts.
[0100] Furthermore, the Qwen-VL 2 multimodal model further includes:
[0101] Regression loss function, the expression of the regression loss function is as follows:
[0102]
[0103] Among them, Area of Overlap: represents the intersection between the detected text box and the standard text box; Area of Union: represents the union between the detected text box and the standard text box.
[0104] In specific implementation, if:
[0105] The coordinates of the upper left corner of frame 1 are (20,20) and the coordinates of the lower right corner are (60,60)
[0106] The coordinates of the upper left corner of frame 2 are (40,40) and the coordinates of the lower right corner are (80,80)
[0107] then:
[0108] Area of Overlap: First, we need to find the coordinates of the intersection area. In this example, the coordinates of the upper left corner of the intersection area are (40,40) and the coordinates of the lower right corner are (60,60). Therefore, the area of the intersection area is (60-40)*(60-40)=400.
[0109] Area of Union: The area of the union is the total area of the two rectangular boxes minus the area of the intersection. The area of box 1 is (60-20)*(60-20)=1600, and the area of box 2 is (80-40)*(80-40)=1600. Therefore, "Area of Union"=1600+1600-400=2800.
[0110] During the model training process, the loss of the conventional generative model can be expressed as:
[0111]
[0112] Here, p is the conditional probability of the t-th word predicted by the model, that is, the probability of the t-th word given the previous t-1 words.
[0113] In the specific implementation, if the pure generation loss leads to inaccurate position recognition of the target box, the "100 and 200" loss and the "100 and 101" loss will be regarded as consistent by the model, because they are two different words, which is obviously unreasonable when recognizing the position.
[0114] To this end, we consider the regression loss in traditional target detection tasks. <box>< / box> The generated result between them will additionally calculate a regression loss, that is, the degree of overlap between the two areas. As shown in Formula 1, the final loss is the sum of the generation loss and the regression loss.
[0115] Specifically, the following example shows the calculation of this loss. If the real data is:
[0116] <ref> Header< / ref> <box> (5,456),(70,590)< / box>
[0117] The model's prediction data is:
[0118] <ref> Title< / ref> <box> (20,420),(70,600)< / box>
[0119] The losses come from:
[0120] Generation loss: The header and title of the ref part are inconsistent
[0121] Regression Loss: Real Data <box> (5,456),(70,590)< / box> , the area is (70-5)×(590-456)=65×134=8710, the predicted data <box> (20,420),(70,600)< / box> , (70-20)×(600-420)=50×180=9000, the intersection area of the two is (20,456), (70,590), the area is (70-20)×(590-456)=50×134=6700, the IoU is 6700 / (8710+9000-6700)=0.441, and the regression loss is 1-IoU=1-0.441=0.559.
[0122] In the specific model prediction process, the prediction is based on the decoding method of the pure generation model, and the output of the next word is performed in sequence, where:
[0123] - <ref>< / ref> The result in will be used as the type of the new content block.
[0124] - <box>< / box> The coordinates in will be used as the location of the content block.
[0125] -The order in which the overall content blocks are generated corresponds to the natural reading order of humans.
[0126] It is understandable that the technical solution provided by the present invention obtains the sales assistant product document information to be identified; the sales assistant product document information to be identified is input into a pre-set multimodal model, and the identified sales assistant product document information is output. It is understandable that the technical solution provided by the present invention utilizes a large multimodal model to simultaneously utilize text and visual information, and simultaneously utilizes text and visual information to perform document structure recognition, with excellent recognition effect; in addition, it can reduce the dependence on the amount of annotated data, and at the same time, the cost of migrating to new scenes is low; an end-to-end model solution can be generated, and the content block position, type, and reading order can be generated at the same time, and the limitations of the generative loss can be compensated by proposing a multi-task loss that combines the generative loss and the region box regression loss.
[0127] See also Figure 3 , Figure 3 The present invention is a schematic diagram showing a composition of a multi-modal large-model sales assistant product document structure recognition system according to an exemplary embodiment, wherein the system comprises:
[0128] An acquisition module 31 is used to acquire the sales assistant product document information to be identified;
[0129] The processing module 32 is used to obtain the sales assistant product document information to be identified, input it into a preset multimodal model, and output the identified sales assistant product document information.
[0130] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.
[0131] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.
[0132] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0133] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0134] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0135] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0136] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.
[0137] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0138] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A multi-modal large-model sales assistant product document structure recognition method, characterized in that: The method comprises: Obtain product document information of the sales assistant to be identified; The sales assistant product document information to be identified is obtained, input into a preset multimodal model, and the identified sales assistant product document information is output.
2. The method according to claim 1, characterized in that The pre-set multimodal model includes: Qwen-VL 2 multimodal model.
3. The method according to claim 1, characterized in that The method of obtaining the sales assistant product document information to be identified, inputting the preset multimodal model, and outputting the identified sales assistant product document information includes: The visual feature information in the sales assistant product document information to be identified is extracted using a preset visual feature extractor; wherein the visual feature includes: coordinate information; The visual feature information in the sales assistant product document information to be identified is extracted using a preset text feature extractor; wherein the text feature includes: tags in HTML language; The visual feature information in the sales assistant product document information to be identified is aligned with the visual feature information in the sales assistant product document information to be identified using a preset visual-text feature alignment module, and the identified sales assistant product document information is output.
4. The method according to claim 2, characterized in that: The Qwen-VL 2 multimodal model includes: Visual feature extractor, text feature extractor, visual-text feature alignment module.
5. The method according to claim 4, characterized in that The visual encoder adopts the VisionTransformer architecture to process the input image and generate a set of image features.
6. The method according to claim 4, characterized in that The visual-text feature alignment module is used to compress the feature sequence of the visual feature information in the sales assistant product document information to be identified extracted by the visual feature extractor, and to align the visual feature information in the sales assistant product document information to be identified with the visual feature information in the sales assistant product document information to be identified, and output the identified sales assistant product document information.
7. The method according to claim 2, characterized in that The Qwen-VL 2 multimodal model also includes: Regression loss function, the expression of the regression loss function is as follows: Among them, Area of Overlap: represents the intersection between the detected text box and the standard text box; Area of Union: represents the union between the detected text box and the standard text box.
8. A multi-modal large model sales assistant product document structure recognition system, applied to the multi-modal large model sales assistant product document structure recognition method according to any one of claims 1 to 7, characterized in that: The system comprises: An acquisition module, used to acquire product document information of a sales assistant to be identified; The processing module is used to obtain the sales assistant product document information to be identified, input it into a preset multimodal model, and output the identified sales assistant product document information.