Information extraction method and related equipment

Abstract

The embodiment of the invention provides an information extraction method and related equipment, and belongs to the technical field of computers. The method comprises the following steps: acquiring multi-modal information containing target information; generating a first cue word based on the cue word template; inputting the multi-modal information into a prediction model, and outputting first target information based on the first cue word; and analyzing the first target information, performing field classification on the first target information, constructing a new first cue word based on the first target information, and returning to the step of inputting the multi-modal information into the prediction model and outputting the first target information based on the first cue word until complete target information is obtained. And outputting all the classified target information. Through continuous updating of the prompt word, complete target information can be extracted, and the integrity of prompt information extraction is facilitated.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an information extraction method and related equipment. Background Technology

[0002] With technological advancements, the amount of information and data generated across all industries is constantly increasing. Information extraction technologies are being widely applied. For example, in some scenarios, relevant departments setting up consumer complaint channels receive a large volume of complaint information, requiring data entry and extraction. Traditional extraction methods rely on manual data entry, which is inefficient, error-prone, and unable to handle massive amounts of data. In recent years, with the development of artificial intelligence and image recognition technologies, more and more organizations have begun to try using technologies such as optical character recognition and natural language processing to automatically extract text content. However, these solutions often suffer from incomplete information extraction.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a method and related equipment for extracting complete information.

[0005] To achieve the above objectives, one aspect of this application proposes an information extraction method. The method includes: acquiring multimodal information containing target information; generating a first prompt word based on a prompt word template; inputting the multimodal information into a prediction model and outputting first target information based on the first prompt word; parsing and classifying the first target information into its fields, constructing a new first prompt word based on the first target information, and returning to the step of inputting the multimodal information into the prediction model and outputting the first target information based on the first prompt word, until complete target information is obtained, and finally outputting all classified target information. This application, through continuous updating of the prompt words, can extract complete target information, which is beneficial to the completeness of the prompt information extraction.

[0006] In some embodiments, the multimodal information includes PDF documents and text, and the step of inputting the multimodal information into a prediction model and outputting first target information based on the first prompt word includes:

[0007] Convert the PDF document into an image;

[0008] The image and text are input into the prediction model, and the first target information is output based on the first prompt word; the first target information includes complainant information.

[0009] In some embodiments, parsing and classifying the first target information, and constructing a new first prompt word based on the first target information, includes:

[0010] Based on the current first target information, query the classification knowledge base, match the current category and level, and use the next level of the current category as the new first prompt word;

[0011] Alternatively, if there is no next level for the current category, do not construct a new first prompt word;

[0012] Alternatively, if no match is found for the current category and level, the previous first prompt word is used as the new first prompt word, and a prediction suggestion is marked. The process then returns to inputting the multimodal information into the prediction model and outputting the first target information based on the first prompt word; the prediction suggestion is unrelated to the current first target information.

[0013] In some embodiments, parsing and classifying the fields of the first target information includes:

[0014] The first target information is parsed, and the fields of the parsed first target information are classified according to the classification knowledge base to obtain structured target information.

[0015] In some embodiments, the classification knowledge base is established through the following steps:

[0016] Extract categories and subcategories from web page elements to create a selection tree;

[0017] The selection tree is separated according to its tree nodes to obtain knowledge bases for different fields.

[0018] In some embodiments, the method further includes:

[0019] All primary target information is categorized to obtain structured target information;

[0020] Convert all first prompt words and first target information into log information and save the log information.

[0021] In some embodiments, converting the PDF document into an image includes:

[0022] The PDF document is rendered page by page using a document rendering engine to obtain images;

[0023] If rendering fails, change the scaling factor and return to the step of rendering the PDF document page by page using the document rendering engine to obtain the image.

[0024] To achieve the above objectives, another aspect of this application provides an information extraction apparatus, the apparatus comprising:

[0025] The acquisition module is used to acquire multimodal information containing target information;

[0026] The prompt word module is used to generate the first prompt word based on the prompt word template;

[0027] The prediction module is used to input the multimodal information into the prediction model and output the first target information based on the first prompt word;

[0028] The extraction module is used to parse and classify the first target information, construct a new first prompt word based on the first target information, return to the step of inputting the multimodal information into the prediction model and outputting the first target information based on the first prompt word, until the complete target information is obtained and all classified target information is output.

[0029] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0030] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0031] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0032] The embodiments of this application include at least the following beneficial effects: This application provides an information extraction method, apparatus, electronic device, storage medium, and program product. The method of this application includes: acquiring multimodal information containing target information; generating a first prompt word based on a prompt word template; inputting the multimodal information into a prediction model and outputting first target information based on the first prompt word; parsing and classifying the first target information into fields, and constructing a new first prompt word based on the first target information; returning to the step of inputting the multimodal information into the prediction model and outputting the first target information based on the first prompt word, until complete target information is obtained, and outputting all classified target information. This application, through continuous updating of the prompt words, can extract complete target information, which is beneficial to the completeness of the prompt information extraction. Attached Figure Description

[0033] Figure 1 This is a flowchart of the information extraction method provided in the embodiments of this application;

[0034] Figure 2 This is a flowchart of another information extraction method provided in the embodiments of this application;

[0035] Figure 3This is a schematic diagram of the structure of web page elements provided in the embodiments of this application;

[0036] Figure 4 This is a tabular illustration of the response process provided in the embodiments of this application;

[0037] Figure 5 This is a schematic diagram of the information extraction device provided in the embodiments of this application;

[0038] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0040] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0041] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0043] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0044] Optical Character Recognition (OCR) is a technology used to convert text content in images into editable text.

[0045] Natural Language Processing (NLP) is a branch of artificial intelligence that studies how to understand and generate human language.

[0046] JavaScript Object Notation (JSON) is a lightweight data-interchange format that is easy for humans to read and machines to parse.

[0047] JSONL: A file format where each line of the JSON Lines format contains a separate JSON object, suitable for scenarios such as logs and data streams;

[0048] vLLM: A high-efficiency local large language model inference engine based on tensor parallelism and a large language model inference framework.

[0049] Application Programming Interface (API) is a standard interface definition for inter-system communication.

[0050] A Universally Unique Identifier (UUID) is a string used to uniquely identify a resource or session.

[0051] Portable Document Format (PDF) is a cross-platform document format that maintains the original layout.

[0052] Hypertext Transfer Protocol (HTTP) is the basic network protocol used for data exchange between clients and servers.

[0053] Gradio: An open-source interactive machine learning demonstration framework that provides the ability to quickly build web interfaces to showcase the effects of AI models;

[0054] FastAPI: An asynchronous, high-performance Python web framework that supports asynchronous request processing and has the ability to automatically generate API documentation;

[0055] Multi-modal Model: An artificial intelligence model that simultaneously supports multiple input formats such as images, text, and speech;

[0056] Structured: Organized in a fixed format (such as JSON) to facilitate subsequent system calls and processing;

[0057] Unstructured: Information with irregular or incomplete structure, without a predefined data model, and inconvenient to represent using a two-dimensional logical table in a database, including all formats of office documents, text, pictures, various reports, images and audio / video information, etc.

[0058] Prompt engineering: a technique that guides large models to output desired content in the expected way by constructing specific language instructions.

[0059] Against this backdrop, there is a growing demand for the development of market supervision information extraction systems.

[0060] With the continuous improvement of the market supervision system, relevant departments need to handle a large number of complaints and reports. These cases are usually submitted in paper or electronic form and saved as PDF files containing full-image content, including structured fields such as the complainant's information, the information of the complainant, the description of the goods or services, and the amount of consumption. Traditional extraction methods rely on manual data entry by relevant personnel, which is inefficient, has a high error rate, and cannot handle massive amounts of data. In recent years, with the development of artificial intelligence and image recognition technology, more and more organizations have begun to try using technologies such as OCR (Optical Character Recognition) and Natural Language Processing (NLP) to automatically extract text content and combine it with knowledge bases for structured output.

[0061] However, most systems currently on the market still have the following problems:

[0062] The recognition accuracy is not high: it has a weak ability to recognize blurry, handwritten, and traditional Chinese characters;

[0063] Incomplete field extraction: Unable to distinguish between "address of the party being complained against" and "envelope address", resulting in confusion of key information;

[0064] Lack of semantic understanding: It only extracts text and cannot determine the field's affiliation based on the context;

[0065] Lack of multi-turn interaction: Lack of the ability to progressively refine fields based on historical question-and-answer sessions;

[0066] Insufficient category matching: A complete category knowledge base has not been established, making it impossible to categorize extracted information into standard fields;

[0067] Privacy protection issues: The work order information in the relevant system involves the privacy of complainants and is not convenient for reasoning using online model services;

[0068] Especially in the "12315 system" for market supervision, many work orders are uploaded as scanned PDF files. These PDF files often contain the following types of information:

[0069] Handwritten information: such as the whistleblower's signature, contact number, etc.;

[0070] Structured tables: such as product name, price, purchase date, etc.;

[0071] Unstructured text descriptions: such as event details, descriptions of requests, etc.;

[0072] Therefore, relying solely on traditional OCR tools to extract plain text from PDFs is insufficient to meet actual business needs. It is essential to introduce visual models to directly parse image content and extract structured output information from images, including but not limited to:

[0073] The complainant's / reporter's name or organization's name;

[0074] The contact number of the person in charge;

[0075] Name of the company / enterprise being complained against;

[0076] The business address of the party being complained about / reported;

[0077] The actual business address of the entity being complained about / reported;

[0078] Name of the goods / services;

[0079] Consumption amount (unit: yuan);

[0080] Reason for reporting;

[0081] Because these fields are presented in various ways across different documents—some appearing in tables, some as handwritten content, and others as printed paragraphs—an intelligent system is needed that can understand image content and output structured information.

[0082] Background information: The closest existing technical solution:

[0083] Currently available technical solutions with similar functionality include:

[0084] ① An OCR-based document structuring system;

[0085] For example, Baidu AI Open Platform and Tencent Cloud OCR interface provide PDF image to text conversion services.

[0086] Functions: Image preprocessing to OCR recognition to text extraction;

[0087] Disadvantages: It cannot distinguish field categories and cannot provide structured output based on business scenarios.

[0088] ② Intelligent question-answering robot system;

[0089] Large models such as Alibaba's Tongyi Qianwen and iFlytek Xinghuo support text and image input and can generate natural language answers.

[0090] Function: Receives text and image questions, processes them through model inference, and outputs the answer;

[0091] Disadvantages: It cannot achieve structured field output and cannot be combined with external knowledge bases for classification and matching;

[0092] ③ Rule-based field extraction system;

[0093] For example, some government systems use regular expressions to match fields with fixed formats.

[0094] Function: Set keyword templates to extract corresponding fields;

[0095] Disadvantages: Poor flexibility, unable to adapt to diverse formats;

[0096] In summary, although various technical methods have been used for document recognition and information extraction, no system has yet been able to simultaneously meet the following conditions:

[0097] Supports image input (especially for extracting scanned images from PDFs);

[0098] Output structured fields;

[0099] Fields are categorized using a classification knowledge base;

[0100] Support a multi-round dialogue mechanism to gradually improve the results;

[0101] Critical information must be hidden.

[0102] Analysis of the main implementation methods and shortcomings of existing technologies:

[0103] In the task of extracting complaint and reporting information in the field of market supervision, the current mainstream technical solutions mainly include the following categories:

[0104] ① OCR-based text recognition system;

[0105] These systems use Optical Character Recognition (OCR) technology to extract text from PDFs or images into plain text. Common examples include Baidu AI Open Platform and Tencent Cloud OCR Interface.

[0106] Problems exist:

[0107] Unable to distinguish field types: Only provides raw text output, lacking the ability to recognize structured information such as "name of the complained party" and "amount spent";

[0108] Poor performance in recognizing blurry images: low recognition rate for handwriting, traditional Chinese characters, and low-resolution images;

[0109] Unable to make judgments based on business semantics: Unable to determine which field category a certain piece of text belongs to based on the context;

[0110] ② A large-scale model-based text and image question-answering system;

[0111] Such systems employ multimodal large models (such as Tongyi Qianwen and Xunfei Xinghuo), which can receive text and image input and generate natural language responses.

[0112] Problems exist:

[0113] The output format is free text: it cannot be directly converted into standard structured output;

[0114] Lack of classification knowledge support: Unable to classify fields in conjunction with the business knowledge base;

[0115] Lack of interactive mechanism: It is impossible to gradually improve field information through multiple rounds of dialogue;

[0116] ③ Field extraction system based on rule matching;

[0117] Some government systems use keyword matching or regular expressions to extract fixed-format fields from text.

[0118] Problems exist:

[0119] Poor flexibility: It can only handle data in a preset format and fails when faced with different layouts or unstructured descriptions;

[0120] Poor scalability: Adding new fields requires manually writing rules, resulting in high development and maintenance costs;

[0121] Unable to adapt to complex document structures: It struggles to handle situations where multiple formats coexist, such as tables, handwriting, and mixed layouts.

[0122] ④ Single prompt word invocation mechanism;

[0123] Some systems attempt to use prompt word engineering to guide the structured output of the model, but only call the prompt word once.

[0124] Problems exist:

[0125] Incomplete information extraction: One-time prompts are insufficient to cover all field requirements;

[0126] Lack of feedback mechanism: Unable to dynamically adjust prompt word content based on model output;

[0127] Difficulty in field categorization: Field path matching was not performed in conjunction with a classification knowledge base;

[0128] This invention is not a simple combination of existing technologies, but rather, based on a full understanding of the limitations of existing technologies, proposes an intelligent information extraction solution for market supervision scenarios. Its core innovation lies in:

[0129] Problem 1: Traditional OCR cannot recognize structured fields;

[0130] The improvement of this invention is to introduce a multimodal model that combines visual recognition and language understanding, which not only extracts the text content in the image, but also determines the field's ownership based on the image layout, field position, and contextual semantics.

[0131] Objective: To automatically identify standard fields such as "name of the party being complained about", "contact number of the party concerned", and "name of goods / services" from images.

[0132] Problem 2: OCR has low recognition rates for handwritten text and blurry images;

[0133] The improvement method of this invention is as follows: image enhancement and multi-scale recognition technology are used to perform high-quality rendering of PDF pages during the image conversion stage, thereby improving the recognition accuracy of subsequent models;

[0134] Objective: To improve the recognition capabilities of scanned documents, handwritten text, and low-resolution images, and to ensure the complete extraction of key fields.

[0135] Problem 3: The output of the large model is free text, which cannot be structured.

[0136] The improvement of this invention is as follows: a standardized prompt word template is designed to guide the model to output results that conform to a predetermined format, and field validation is performed in conjunction with a post-processing module;

[0137] The purpose is to ensure the integrity and consistency of the output fields each time, so as to facilitate subsequent system integration and use.

[0138] Problem 4: Lack of classification knowledge leads to chaotic field categorization;

[0139] The improved method of this invention is to construct an adjacency list dictionary and use the structured data of web pages to establish a parent-child relationship tree for field path matching and classification suggestions;

[0140] The corresponding objective is to help the model select the most suitable field classification label from multiple possible options, thereby improving the accuracy and usability of the output data.

[0141] Problem 5: Single prompt word call cannot complete field information;

[0142] The improvement of this invention is as follows: a multi-round question-and-answer mechanism is designed, and new prompt words are constructed based on the initial output results of the model to guide the model to gradually refine the field content;

[0143] The objective is to continuously correct and supplement field information through multiple rounds of interaction in a single query, and finally output complete structured data.

[0144] In summary, the core value of this invention lies in:

[0145] This is the first time that a multimodal large model has been applied to the field of market supervision, solving the problem of extracting Chinese fields from PDF images after scanning paper work orders;

[0146] By introducing mechanisms such as image enhancement, prompt word templates, multi-turn question answering, and classification knowledge base, the shortcomings of existing OCR, large model question answering, and rule matching technologies are effectively made up for.

[0147] It implements an end-to-end structured information extraction process, is applicable to various document formats (including handwritten text, tables, and mixed paragraph content), and has good practicality and scalability;

[0148] Therefore, this invention is a technical solution with practical application value, which can significantly improve the information processing efficiency and automation level of market supervision departments.

[0149] The information extraction method provided in this application relates to the field of computer technology. The information extraction method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the information extraction method, but is not limited to the above forms.

[0150] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0151] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0152] Figure 1 This is an optional flowchart of the information extraction method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S100 to S400.

[0153] Step S100: Obtain multimodal information containing target information;

[0154] Step S200: Generate the first prompt word based on the prompt word template;

[0155] Step S300: Input the multimodal information into the prediction model and output the first target information based on the first prompt word;

[0156] Step S400: parse and classify the first target information, construct a new first prompt word based on the first target information, return to the step of inputting the multimodal information into the prediction model and outputting the first target information based on the first prompt word, until complete target information is obtained, and output all classified target information.

[0157] In some embodiments, this application can process multimodal information, including images, text, etc. The prompt word template can be set according to the domain. This application parses and categorizes the first target information obtained from each first prompt word, and then integrates all the first target information to obtain structured, categorized target information.

[0158] In some embodiments, the multimodal information includes PDF documents and text, and the step of inputting the multimodal information into a prediction model and outputting first target information based on the first prompt word includes:

[0159] Convert the PDF document into an image;

[0160] The image and text are input into the prediction model, and the first target information is output based on the first prompt word; the first target information includes complainant information.

[0161] In some embodiments, parsing and classifying the first target information, and constructing a new first prompt word based on the first target information, includes:

[0162] Based on the current first target information, query the classification knowledge base, match the current category and level, and use the next level of the current category as the new first prompt word;

[0163] Alternatively, if there is no next level for the current category, do not construct a new first prompt word;

[0164] Alternatively, if no match is found for the current category and level, the previous first prompt word is used as the new first prompt word, and a prediction suggestion is marked. The process then returns to inputting the multimodal information into the prediction model and outputting the first target information based on the first prompt word; the prediction suggestion is unrelated to the current first target information.

[0165] This application matches the first target information with a classification knowledge base, using the value corresponding to the next level of the current category as the new first prompt word. If no next level exists, no new first prompt word is constructed, and all the currently obtained first target information is used as the complete target information. If no category or level is matched, the previous first prompt word is returned, and the prediction model is alerted through prediction suggestions that the current first target information is not the desired prediction result.

[0166] In some embodiments, parsing and classifying the fields of the first target information includes:

[0167] The first target information is parsed, and the fields of the parsed first target information are classified according to the classification knowledge base to obtain structured target information.

[0168] In some embodiments, the classification knowledge base is established through the following steps:

[0169] Extract categories and subcategories from web page elements to create a selection tree;

[0170] The selection tree is separated according to its tree nodes to obtain knowledge bases for different fields.

[0171] In some embodiments, the method further includes:

[0172] All primary target information is categorized to obtain structured target information;

[0173] Convert all first prompt words and first target information into log information and save the log information.

[0174] In some embodiments, converting the PDF document into an image includes:

[0175] The PDF document is rendered page by page using a document rendering engine to obtain images;

[0176] If rendering fails, change the scaling factor and return to the step of rendering the PDF document page by page using the document rendering engine to obtain the image.

[0177] The method provided in this application will be described in detail below with specific application examples:

[0178] Reference Figure 2 The architecture described in the text explains the system's component modules:

[0179] ① Image conversion module;

[0180] Function: Converts PDF documents into high-quality image files for easier subsequent model recognition.

[0181] Implementation: The document rendering engine renders the PDF content page by page as an image format, supporting multiple resolution settings and page range control.

[0182] Key technical points:

[0183] Supports scaling factor to control image sharpness;

[0184] An automatic retry mechanism handles rendering failures by changing the scaling factor to retry converting PDF pages to image files.

[0185] Supports extraction of a specified page range.

[0186] ② Multimodal large model service module;

[0187] Function: Deploy large models with image and text understanding capabilities, receive mixed image and text input, and output structured field information.

[0188] Implementation: By loading a multimodal model locally, an inference service interface is built to support the processing of multiple concurrent requests.

[0189] Key technical points:

[0190] Supports mixed text and image input;

[0191] Dialogue state management;

[0192] Supports both streaming and non-streaming output.

[0193] ③ Prompt word templates and multi-turn question-and-answer module;

[0194] Function: Guides the model step-by-step to output structured field information through prompt words.

[0195] Implementation method: Initial prompt words are generated based on preset field templates, and the prompt content is continuously updated based on model feedback to form a multi-round question and answer mechanism.

[0196] Key technical points:

[0197] The initial prompt word definition field requires extracting complainant information from the image file converted from the target PDF, such as: complainant / reporter's name or organization name, contact number of the party concerned, name of the company / enterprise being complained against, name of the goods / services, reason for the complaint, etc.

[0198] The model answers the query to the categorized knowledge base;

[0199] Based on the answer from the previous step, continue searching for sub-options, construct new prompts, and call the model again. For example, when it is necessary to retrieve the category of the complained product or service, when the model selects the product, there will be sub-options [food, medicine, cosmetics, clothing, ...].

[0200] The multi-round question-and-answer session ends when there are no more sub-options in the current selection tree, and the final selection result is obtained, such as: Goods-Food-Meat-Cured Meat-Starch Sausage.

[0201] Detailed explanation of the role of prompt word engineering:

[0202] In this invention, prompt word engineering serves as a bridge connecting user intent and model output, and its role is mainly reflected in the following aspects:

[0203] Standardized output format: By pre-setting field descriptions and structured format requirements, we ensure that the model output meets business needs;

[0204] Enhance semantic understanding capabilities: By designing contextual cue words, the model can better understand the attribution of fields in image content;

[0205] Multi-round interaction mechanism: Construct new prompt words based on the model output to guide the model to gradually improve field information;

[0206] Assisting in classification decisions: By combining the path options provided by the classification knowledge base, candidate suggestions are added to the prompt words to improve the accuracy of field classification;

[0207] Fault tolerance and error correction mechanism: When the model output is incomplete or incorrect, the problem can be corrected by prompt words to improve the stability of the output. For example, when the model cannot select a matching option in the current sub-option tree, the AI ​​model is prompted to make an incorrect selection by going back to the previous level of the current sub-option tree, so that the model can reselect in the previous level of options and remind that the previous selection caused it to be unable to continue to select, thereby avoiding entering the same sub-option tree again.

[0208] ④ Categorized knowledge base module;

[0209] Functionality: Builds a hierarchical relational database that supports field classification matching.

[0210] Implementation method: Construct a parent-child relationship tree based on the structured content of the webpage, and store it in an extensible data format for reference during model inference.

[0211] Key technical points:

[0212] Using HTML parsing to construct an adjacency list, we extract all options and their corresponding sub-option fields from webpage elements and recombine them into a selection tree. The original HTML webpage elements typically look like this: Figure 3 The structure shown;

[0213] You can continue to dig down the child option tree nodes custom-tree-node by following the relationship from custom-tree-node to el-tree-node__children to treeitem, until there is no next level;

[0214] The system employs a multi-level classification system. The option tree mined in the previous step has multiple root nodes, corresponding to different dropdown menu contents on the webpage. This means there are multiple selection trees and their corresponding sub-option tree branches. In actual webpage operation, each dropdown menu contains the complete selection path after the user makes their choice. Therefore, this system needs to separate the multiple root nodes and their corresponding tree node branches and store them in different knowledge bases. These will be used in step ③ to construct selection trees for different fields within the prompt word template.

[0215] ⑤ Structured output and logging module;

[0216] Function: Saves the extracted results as structured data and records the operation process.

[0217] Implementation method: Persistent storage using structured data format, while recording log information of the interaction process.

[0218] Specific explanation: During a request processing session, this system needs to extract information from the image and complete the values ​​of required fields. The most crucial aspect is continuously selecting complete paths from multiple choice trees. To allow for backtracking of the model's path selections and intermediate options for each field during process ③, and to facilitate developers or users checking the reliability of the AI ​​model's responses, the system saves the complete content of the constructed prompts and the model's corresponding responses as a table file at the end of each request. For example... Figure 4 As shown, the interaction process is presented in a table format.

[0219] Use blank lines to separate each field value that needs to be extracted, so that the question and answer content in each consecutive line is a complete pathfinding history of the selection tree path.

[0220] The system workflow can be divided into the following stages:

[0221] Image preparation stage: Convert user-uploaded PDFs or image files into standard image formats to ensure the model can accurately recognize them.

[0222] Initial prompt word invocation: Uses a preset prompt word template to guide the model to output preliminary structured fields.

[0223] Field analysis and classification matching: Parse the model output and classify the fields using a classification knowledge base;

[0224] Iterative optimization of prompt words: Construct new prompt words based on the results of the previous round to guide the model to further refine the field content.

[0225] Structured output and logging: Save the final extracted results in a formatted output (such as JSON) and log the interaction process.

[0226] The core of this invention lies in constructing a structured extraction system and method for market supervision information based on a multimodal large model. Through key technologies such as image recognition, prompt word engineering, and linkage of classification knowledge base, it realizes the automated extraction and structured output of complaint and reporting information from PDF page content or directly from specified multiple image files.

[0227] ① Image preprocessing and conversion mechanism based on visual models;

[0228] Technical Description:

[0229] This invention employs image enhancement and multi-scale rendering techniques to convert the original PDF document page by page into high-quality image files, ensuring that subsequent models can accurately recognize handwritten text, blurred images, traditional Chinese characters, and other content.

[0230] What problem is being solved?

[0231] Many of the complaints and reports received by relevant departments are scanned copies or PDF files converted from paper documents. The content may be handwritten, blurry, or poorly formatted. If the text is read directly, the recognition rate is very low.

[0232] How to solve:

[0233] We designed a method to convert each page of a PDF into a clear image, so that the system can extract information more accurately when viewing the images later.

[0234] Optimize the image quality so that the model can clearly see complex content such as handwriting and blurry text;

[0235] If the conversion fails, the system will automatically attempt to reprocess it to ensure that it does not "get stuck".

[0236] ② Multimodal large model service interface design and support for mixed text and image input;

[0237] Technical Description:

[0238] This invention deploys a local multimodal large model (such as Qwen2-VL), builds an inference service interface, supports mixed input of images and text, and returns structured output.

[0239] What problem is being solved?

[0240] Ordinary OCR can only recognize a bunch of text, but it doesn't know what each character means. We hope that AI can "see" images and understand which parts contain key information such as phone numbers, company names, and addresses, just like humans can.

[0241] How was it resolved?

[0242] We used an advanced multimodal model that not only "sees" images but also works in conjunction with prompt word templates (which tell it what information to look for) to output the field information we want, such as:

[0243] Complainant's name;

[0244] The name of the company being complained about;

[0245] Product Name;

[0246] Amount spent;

[0247] Key points include:

[0248] A mechanism that allows simultaneous input of images and prompts;

[0249] How does the AI's response gradually refine the information during multiple interactions?

[0250] How does the system manage requests from multiple users and prevent confusion? Different users agree on different dynamic random codes with the system to manage their own dialogue context history.

[0251] ③ A multi-round question-and-answer mechanism guided by prompt words;

[0252] Technical Description:

[0253] This invention introduces Prompt Engineering as one of its core technologies. Through standardized prompt word templates and a dynamic construction mechanism, it guides the model to gradually improve field information and form a multi-round question-and-answer process.

[0254] What problem is being solved?

[0255] AI may not be able to convey all the information at once, just as a student needs repeated questioning from a teacher to answer questions better. We need a mechanism to help AI be continuously "reminded" until it conveys all the necessary information.

[0256] How was it resolved?

[0257] We set up a "standard question bank" for the AI—initial prompts—telling it which fields to extract. Then, based on its answers, we generate new questions to guide it to continue supplementing information until the result is complete.

[0258] For example:

[0259] It's like taking an exam. The teacher first asks you, "What is this person's phone number?" After you answer, the teacher then asks, "And what is their address?" Through round after round of questions, you finally get the complete answer.

[0260] Key points include:

[0261] Contents of the prompt template;

[0262] A method for constructing new questions based on AI's answers;

[0263] How to determine when the AI ​​has provided a complete answer and stop asking questions;

[0264] Add classification suggestions (i.e. prediction suggestions) to the prompts to help the AI ​​make choices.

[0265] ④ The construction and application mechanism of the classification knowledge base;

[0266] Technical Description:

[0267] This invention constructs a complete classification knowledge base for field categorization and matching. This knowledge base builds a parent-child relationship tree based on the structured content of web pages and stores it in a structured format for use during model inference.

[0268] What problem is being solved?

[0269] In market supervision, there are many fixed options, such as the major categories of reasons for reporting, such as "quality problems" and "false advertising". We cannot let AI play freely and choose at will; we must select the most appropriate one from the existing regulations.

[0270] How was it resolved?

[0271] We extract structured business knowledge from web pages and organize it into a "parent-child relationship table," which represents "what child categories are under a parent category," for example:

[0272] Parent level: Product issue;

[0273] Subcategories: Quality issues, packaging discrepancies, false advertising;

[0274] When outputting fields, AI will refer to this database to make more accurate choices.

[0275] For example:

[0276] This is like equipping AI with a "standard answer library," so that when it answers a question, it prioritizes selecting answers from the library rather than making them up itself.

[0277] Key points include:

[0278] The construction method of the categorized knowledge base (based on webpage extraction);

[0279] Field path matching algorithm;

[0280] The method of embedding category suggestions into prompt words;

[0281] The knowledge base is dynamically updated and expanded by having a background robot periodically access the platform's webpage to obtain a list of available items and sub-items, which is then updated in the local knowledge base.

[0282] Compared to mainstream OCR recognition systems, large-scale image-text question-answering systems, and rule-based field extraction methods currently on the market, this invention achieves significant improvements in several key aspects. Firstly, in the image processing stage, traditional OCR tools can only extract text but cannot distinguish field categories, and their ability to recognize handwritten text and blurred images is weak. This invention improves the accuracy of recognizing scanned documents and handwritten content by introducing image enhancement and multi-scale rendering technologies, and converts PDFs into high-quality images, laying the foundation for subsequent structured extraction.

[0283] Secondly, regarding information extraction methods, while existing large-scale models can understand text and image content, their output is usually in the form of free text, making it difficult to directly use in business systems. This invention guides AI output standardization through prompt word engineering and combines it with a classification knowledge base to achieve field categorization, giving the output results good structured characteristics and facilitating subsequent system calls. Simultaneously, we have designed a multi-round question-and-answer mechanism, allowing AI to continuously optimize its output based on previous responses, gradually improving field content and compensating for the incomplete information provided by a single prompt word call.

[0284] Furthermore, considering the diverse types, formats, and semantics of fields in market supervision scenarios, this invention introduces a classification knowledge base to assist in field matching, ensuring that AI output conforms to business specifications and avoiding misjudgments or omissions. The entire system also supports log recording and automatic temporary file cleanup, guaranteeing system stability and traceability. These improvements make this invention not only highly innovative at the technical level but also demonstrate a higher level of intelligence and practical value in real-world applications.

[0285] Please see Figure 5 This application also provides an information extraction device that can implement the above-described method. The device includes:

[0286] The acquisition module 610 is used to acquire multimodal information containing target information;

[0287] Prompt word module 620 is used to generate the first prompt word based on the prompt word template;

[0288] Prediction module 630 is used to input the multimodal information into the prediction model and output the first target information based on the first prompt word;

[0289] Extraction module 640 is used to parse and classify the first target information, construct a new first prompt word based on the first target information, return to the step of inputting the multimodal information into the prediction model and outputting the first target information based on the first prompt word, until the complete target information is obtained and all classified target information is output.

[0290] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0291] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0292] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0293] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0294] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0295] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application.

[0296] The input / output interface 903 is used to implement information input and output;

[0297] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0298] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0299] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0300] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0301] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0302] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0303] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0304] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0305] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0306] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0307] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0308] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0309] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0310] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0311] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0312] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0313] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0314] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0315] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An information extraction method, characterized in that, The method includes the following steps: Obtain multimodal information containing target information; Generate the first prompt word based on the prompt word template; The multimodal information is input into the prediction model, and the first target information is output based on the first prompt word; The first target information is parsed and categorized into fields, and a new first prompt word is constructed based on the first target information. The process of inputting the multimodal information into the prediction model and outputting the first target information based on the first prompt word is repeated until complete target information is obtained, and all categorized target information is output.

2. The method according to claim 1, characterized in that, The multimodal information includes PDF documents and text. The step of inputting the multimodal information into a prediction model and outputting first target information based on the first prompt word includes: Convert the PDF document into an image; The image and text are input into the prediction model, and the first target information is output based on the first prompt word; the first target information includes complainant information.

3. The method according to claim 1, characterized in that, The step of parsing and classifying the first target information, and constructing a new first prompt word based on the first target information, includes: Based on the current first target information, query the classification knowledge base, match the current category and level, and use the next level of the current category as the new first prompt word; Alternatively, if there is no next level for the current category, do not construct a new first prompt word; Alternatively, if no match is found for the current category and level, the previous first prompt word is used as the new first prompt word, and a prediction suggestion is marked. The process then returns to inputting the multimodal information into the prediction model and outputting the first target information based on the first prompt word; the prediction suggestion is unrelated to the current first target information.

4. The method according to claim 1, characterized in that, The parsing and field classification of the first target information includes: The first target information is parsed, and the fields of the parsed first target information are classified according to the classification knowledge base to obtain structured target information.

5. The method according to claim 3 or claim 4, characterized in that, The classification knowledge base is established through the following steps: Extract categories and subcategories from web page elements to create a selection tree; The selection tree is separated according to its tree nodes to obtain knowledge bases for different fields.

6. The method according to claim 1, characterized in that, The method further includes: All primary target information is categorized to obtain structured target information; Convert all first prompt words and first target information into log information and save the log information.

7. The method according to claim 2, characterized in that, The process of converting the PDF document into an image includes: The PDF document is rendered page by page using a document rendering engine to obtain images; If rendering fails, change the scaling factor and return to the step of rendering the PDF document page by page using the document rendering engine to obtain the image.

8. An information extraction device, characterized in that, The device includes: The acquisition module is used to acquire multimodal information containing target information; The prompt word module is used to generate the first prompt word based on the prompt word template; The prediction module is used to input the multimodal information into the prediction model and output the first target information based on the first prompt word; The extraction module is used to parse and classify the first target information, construct a new first prompt word based on the first target information, return to the step of inputting the multimodal information into the prediction model and outputting the first target information based on the first prompt word, until the complete target information is obtained, and output all classified target information.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images