Zero-sample graphics and text information extraction method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image-text information extraction, in particular to a zero-sample image-text information extraction method and system, electronic equipment and a storage medium. The method comprises the following steps: collecting image-text data containing images and corresponding characters, inputting the images into a business inference model to obtain preliminary target business information, generating image inference description by utilizing a visual big language model, and inputting the preliminary target business information, the image inference description and the character data through the big language model, and generating final target business information in combination with the retouching cue word. According to the method, large-scale sample labeling is not needed, the zero sample requirements of various professional fields can be rapidly met, the problems that a traditional model is insufficient in accuracy and poor in adaptability are solved, the information extraction quality is improved, and reliable support is provided for downstream application.

Description

Technical Field

[0001] This invention relates to the field of image and text information extraction technology, and in particular to a zero-sample image and text information extraction method, system, electronic device and storage medium. Background Technology

[0002] With the rapid development of the digital economy, industries such as industrial production, commodity manufacturing, and e-commerce services have accumulated massive amounts of digital image and text data. Accurately extracting target business information such as product attributes from these data has become a core requirement for supporting downstream applications such as user product selection, product recommendation, and intelligent question answering. Visual language models, as a key technology for multimodal information processing, already possess the ability to recognize mixed data such as text and charts in images and extract information based on user-instructed prompts, providing a technological foundation for image and text information extraction. However, existing visual language models rely excessively on their native general capabilities and generic prompts in practical applications, lacking targeted adaptation to specific business scenarios. This makes it difficult to accurately capture specific information conforming to business standards when processing image and text data in zero-sample scenarios, often resulting in extraction results with issues such as attribute recognition bias, omission of key information, and excessive redundant information.

[0003] The aforementioned technical deficiencies directly hinder performance improvements in downstream applications: in e-commerce scenarios, inaccurate product attribute extraction leads to a mismatch between recommended results and user needs, impacting the user shopping experience; in industrial production, vague product attribute information may cause supply chain management chaos or product selection errors. Traditional methods have failed to effectively resolve the compatibility contradiction between general models and specialized business scenarios, and cannot meet the high requirements for accuracy and reliability in extracting target business information under zero-sample conditions.

[0004] Therefore, there is an urgent need for a technical solution that can overcome the limitations of general models and achieve accurate conversion of text and image information into target business information in zero-sample business scenarios, so as to solve the core technical problems of insufficient accuracy and poor adaptability of existing visual language models in professional field information extraction. Summary of the Invention

[0005] The purpose of this invention is to provide a zero-sample image and text information extraction method, system, electronic device, and storage medium to solve the problems of insufficient accuracy in information extraction in professional fields and poor adaptability to specific business scenarios of existing visual language models.

[0006] The technical solution of this invention is: a zero-sample image and text information extraction method, comprising: Collect image and text data; the image and text data includes image data and corresponding text data, wherein the text data includes the text corresponding to the image, title and description; The image data is input into the business reasoning model to obtain preliminary target business information; Image data is input into a visual large language model for reasoning to obtain an image reasoning description; The large language model takes preliminary target business information, image reasoning descriptions, and text data as input, and combines them with polishing prompts to generate the final target business information.

[0007] Preferably, the business reasoning model includes a pre-trained visual encoder and a trained dimensionality reduction decoding model.

[0008] Preferably, the pre-trained visual encoder is a pre-trained visual encoding module based on the CLIP architecture, used to receive image data and extract high-dimensional visual features; the high-dimensional visual features include the image's texture, shape, color, target layout, and scene association information.

[0009] Preferably, the training process of the trained dimensionality reduction decoding model includes: Collect text and image data and obtain corresponding target business information; A visual large language model is used to reason about image data to obtain image reasoning descriptions; The parameters of the pre-trained text encoder are frozen, and the image reasoning description, image and text data and corresponding target business information are used as input to the text encoder. After the text encoder calculates, the text features are obtained. Using text features as input and target business information as labels, a dimensionality reduction decoding model is trained; wherein, the dimensionality reduction decoding model is composed of a linear projector and a text decoder connected in series.

[0010] Preferably, the pre-trained text encoder is a pre-trained text encoding module based on the CLIP architecture. Its parameters are frozen during the training phase. It is used to receive image reasoning descriptions, image and text data and corresponding target business information, and to calculate text features through semantic encoding.

[0011] Preferably, the polishing prompts are used to guide the large language model to perform accuracy verification, redundant information removal, and format standardization processing on the initial target business information.

[0012] On the other hand, this application also discloses a zero-sample image and text information extraction system, including: The data acquisition module is used to collect image and text data; the image and text data includes image data and corresponding text data, wherein the text data includes the text corresponding to the image, title and description; The preliminary information extraction module is used to input image data into the business reasoning model to obtain preliminary target business information; the business reasoning model includes a visual encoder and a trained dimensionality reduction decoding model. The image description generation module is used to input image data into the visual big language model for reasoning and obtain image reasoning descriptions. The information optimization and generation module is used to generate the final target business information by taking preliminary target business information, image reasoning descriptions and text data as inputs through a large language model and combining them with polishing prompts.

[0013] Preferably, the system further includes a model training module, which is used to train the dimensionality reduction decoding model, including: The sample processing unit is used to collect sample image and text data and obtain corresponding target business information. It uses a visual big language model to reason about the sample image data to obtain a sample image reasoning description. The feature extraction unit is used to select a pre-trained text encoder, freeze its parameters, and input the sample image inference description, sample text data, and target business information into the text encoder to obtain text features. The model training unit is used to train a dimensionality reduction decoding model consisting of a linear projector and a text decoder connected in series, using text features as input and target business information as labels.

[0014] On the other hand, this application further discloses an electronic device, including: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement any of the zero-sample image and text information extraction methods described above.

[0015] On the other hand, this application further discloses a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements any one of the zero-sample image and text information extraction methods described above.

[0016] Compared with the prior art, the advantages of the present invention are: (1) Based on the complete collection of image and text data, it comprehensively includes images and corresponding text, titles, descriptions and other information to avoid the deficiency of a single information source. Through the business reasoning model, the visual information of the images is transformed into a preliminary attribute framework that fits the business needs. There is no need for large-scale labeled samples, which breaks the limitation of general models in adapting to professional scenarios and alleviates the pain point of adaptability in zero-sample scenarios.

[0017] (2) Introduce a visual big language model to parse images, transforming visual details such as textures and scenes into semantic descriptions to uncover implicit business information. By integrating preliminary target business information, image descriptions and original text through the big language model, verification and redundancy removal are completed under the guidance of polishing prompts, accurately correcting attribute deviations, filling in key information gaps, and solving the problem of insufficient accuracy in extraction by traditional methods.

[0018] (3) During the model training phase, the model is adapted to the image and text database and corresponding target business information tags based on specific business scenarios. It can quickly adapt to the image and text information extraction needs of multiple fields such as industrial production, commodity manufacturing, and e-commerce services without relying on a large number of labeled samples. This solves the pain point of poor adaptability of traditional methods to professional scenarios and significantly improves the scenario transfer capability of the technical solution. Attached Figure Description

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart of a zero-sample image and text information extraction method according to the present invention; Figure 2 This is a flowchart of the training process for the dimensionality reduction decoding model described in this invention; Figure 3 This is a schematic diagram of the structure of a zero-sample image and text information extraction system according to the present invention. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to specific embodiments: This application is applied in fields such as industry, manufacturing, and e-commerce, addressing the need for product attribute extraction from massive digital image documents, and adapting to scenarios such as user product selection, product recommendation, and intelligent question answering. Through cross-modal coding multi-scenario adaptation training, multi-source information fusion, and large-scale model inference, it solves the problems of insufficient accuracy and poor scenario adaptability in professional domain information extraction of traditional visual language models. It can quickly output accurate structured attribute information in zero-shot scenarios, improving the performance of downstream applications and user experience.

[0021] like Figure 1 As shown, a zero-sample image and text information extraction method includes: S1. Collect image and text data; image and text data includes image data and corresponding text data, where text data includes the text corresponding to the image, title and description.

[0022] Specifically, data is integrated with industrial production systems, e-commerce platform merchant back-ends, and manufacturer document management systems to collect or acquire image and text data in target scenarios in batches or in real time. During the collection process, the data undergoes format verification and preprocessing, removing invalid data such as damaged images, unparseable encrypted files, blank text, and garbled text. Images are standardized in size, and text data is deduplicated, segmented, and punctuated to ensure consistent data format.

[0023] In one implementation, for furniture products, the image data includes the main product image, detailed real-life photos, specification diagrams, and usage scenario images; the corresponding text covers structured or semi-structured text fragments from the product details page, such as parameter panel text, material composition descriptions, and process technology analysis; the title is the core identifier name set when the product is listed, containing key information such as brand, core attributes, and product type; the description is a multi-dimensional detailed introduction filled in by the merchant, including design concept, functional features, material source, target audience, and maintenance methods. After collection, a structured database is established according to the association rules between image IDs and text data indexes, using a key-value pair storage method to achieve rapid association and retrieval of images and corresponding text data. Metadata such as data collection time, source channel, and data quality score is recorded to support subsequent data traceability and quality optimization.

[0024] S2. Input the image data into the business reasoning model to obtain preliminary target business information.

[0025] Specifically, the business reasoning model is a specialized model trained and optimized for specific scenarios. Its core function is to transform the visual information of images into attributed information that initially aligns with business requirements. During model execution, the preprocessed input image undergoes deep feature mining to extract high-dimensional visual feature vectors, including color distribution, texture features, morphological structure, and local detail features, ensuring that the features cover visual dimensions strongly correlated with business attributes. This high-dimensional feature vector is then mapped to a low-dimensional feature space that fits the business attributes. After semantic mapping and probability prediction, preliminary target business information is output. This preliminary information already possesses a clear framework of business attribute categories, such as core dimensions like material, style, type, and form in a furniture scene, and the attribute values ​​are highly correlated with the image's visual features, providing a high-quality foundational framework for subsequent optimization stages.

[0026] In one implementation, after inputting the image and text data of a solid wood dining table into the business inference model, the model first captures visual features such as the wood grain texture of the tabletop, the design of the table legs, and the overall color tone, generating a high-dimensional feature vector. Then, through preset attribute mapping rules, the feature vector is transformed into the corresponding attribute prediction results. The final output of preliminary target business information covers core dimensions such as material, style, type, shape, and color, such as "Material: Solid wood, Style: Chinese, Type: Dining table, Shape: Rectangular, Color: Walnut color". This result has initially met the basic requirements for furniture attribute extraction, but there may be hidden attributes that are not covered or require further verification.

[0027] S3. Input the image data into the visual big language model for reasoning to obtain the image reasoning description.

[0028] Specifically, the visual big language model performs comprehensive visual understanding of the input image data, identifies key information such as target objects, appearance attributes, and scene associations in the images, and transforms this information into natural, fluent, and logically complete text descriptions to ensure coverage of core visual details related to the target business.

[0029] The visual big language model possesses cross-modal understanding and high-precision semantic generation capabilities. Its reasoning process is divided into two stages: visual parsing and language generation. In the visual parsing stage, the model comprehensively analyzes the global scene, local details, and relationships between the target subject and the image through a multi-scale visual attention mechanism. It accurately identifies the appearance, material texture, color style, structural characteristics, and scene-related information of the target object in the image, while also capturing the implicit relationships between the target object and its surrounding environment. In the language generation stage, based on the parsed visual semantic information, the model generates logically coherent, detailed, and key-point text descriptions according to the logic of natural language expression. The descriptions strictly focus on core information related to the target business attributes, avoiding irrelevant and redundant expressions, ensuring accurate semantic support for subsequent information fusion.

[0030] In one implementation, after inputting an image of a solid wood dining table into a visual big language model, the model first parses out visual information such as the natural walnut wood grain of the tabletop, the square solid wood structure of the legs, the simple overall design without unnecessary decoration, the rounded corners of the tabletop, its placement in a Chinese-style restaurant space, and its pairing with Chinese-style dining chairs. Subsequently, it generates an image reasoning description, which fully covers the core visual information related to furniture attribute extraction, such as material, style, type, shape, structure, and usage scenario, providing rich semantic basis for the verification and supplementation of initial target business information.

[0031] S4. The large language model takes preliminary target business information, image reasoning descriptions and text data as input, and combines them with polishing prompts to generate the final target business information.

[0032] Specifically, the large language model first integrates multi-source information such as preliminary target business information, image reasoning descriptions, and text data. Then, according to the rules set by the polishing prompts, it verifies, filters, and organizes the integrated information, removes invalid and contradictory information, supplements key missing information, and finally outputs structured target business information that meets business standards.

[0033] Specifically, the large language model employs a multi-source information fusion and refined optimization mechanism to generate the final target business information in three steps. The first step is multi-source information alignment and fusion. Specifically, it performs semantic parsing and alignment processing on the initial target business information, image reasoning descriptions, and text data. Through entity recognition, keyword extraction, and semantic similarity calculation, it establishes attribute association mappings between the three types of data, forming a comprehensive semantic pool that integrates multi-dimensional information. The second step is refined optimization guided by prompt words. Specifically, it presets customized polishing prompt words based on the target business scenario. The prompt words clearly include the attribute extraction range, information quality requirements, format specifications, and optimization rules. Guided by the prompt words, the model performs attribute-by-attribute verification, supplements missing attributes, and removes redundant information from the information in the comprehensive semantic pool. The third step is structured output. Specifically, it organizes and sorts the optimized attribute information according to the format specifications specified by the prompt words, generating a clearly hierarchical, uniformly formatted, and directly usable structured final target business information for downstream applications.

[0034] In one implementation, polishing prompts are set. The large language model integrates the initial target business information, image reasoning descriptions, and text data: it verifies and confirms the accuracy of attribute values ​​such as material and style; it supplements the attributes required by the prompts from the title and description, and removes redundant information required by the prompts from the text data; finally, it outputs structured target business information to meet the business standards for furniture product attribute extraction and the needs of downstream applications.

[0035] In summary, this application ensures comprehensive and complete information sources by collecting complete image and text data, including both images and corresponding text. A core attribute framework is rapidly built using a business reasoning model, and a visual language model is used for deep image analysis to supplement key visual semantics. Further verification and optimization are achieved by fusing multi-source data using a large-scale language model. This significantly improves the accuracy of professional domain information extraction in zero-sample scenarios, effectively reducing attribute recognition bias and the omission of key information. This solution requires no large-scale sample annotation, allowing for rapid adaptation to the needs of multiple fields such as industry, manufacturing, and e-commerce. It efficiently addresses the pain point of poor scenario adaptability of traditional models, ultimately outputting structured, high-quality target business information. This provides reliable data support for downstream applications such as product selection, recommendation, and intelligent question answering, comprehensively improving application service performance and user interaction experience.

[0036] Based on the above implementation, since the business reasoning model transforms the visual information of an image into attributed information that initially fits the business requirements, in an exemplary embodiment, the business reasoning model may include a pre-trained visual encoder and a trained dimensionality reduction decoding model.

[0037] The pre-trained visual encoder is a pre-trained visual encoding module based on the CLIP architecture, which is used to receive image data and extract high-dimensional visual features. The high-dimensional visual features include the image's texture, shape, color, target layout, and scene association information.

[0038] Specifically, the pre-trained visual encoder employs a CLIP-based visual encoding module. This module, pre-trained on large-scale image and text data, possesses powerful cross-modal visual feature extraction capabilities, accurately capturing diverse visual information related to business attributes within images. Upon receiving image data, it uses multi-scale convolution, attention mechanisms, and other techniques to progressively analyze the image's texture details, geometric shapes, color distribution, target layout, and scene relationships. This information is then integrated into a high-dimensional visual feature vector, providing valuable features for subsequent dimensionality reduction decoding.

[0039] In one implementation, for a furniture product scenario, a real-life image of a Nordic-style solid wood sofa is input into the visual encoder. The encoder identifies deep, high-dimensional features such as sofa texture, overall shape, and geometric shape, and outputs a high-dimensional visual feature vector containing diverse information, fully covering the visual basis related to the sofa's core attributes such as material, style, and form. After receiving this feature vector, the trained dimensionality reduction decoding model can quickly map and decode the corresponding preliminary target business information.

[0040] Specifically, such as Figure 2 As shown, the training process of the dimensionality reduction decoding model includes: S201. Collect graphic and textual data and obtain the corresponding target business information.

[0041] Specifically, the training of the dimensionality reduction decoding model focuses on scenario adaptation to ensure that the model can accurately learn the mapping relationship between text and image data and target business information. First, text and image data in the target scenario are collected, and standard target business information that has been manually annotated or confirmed by the business system is obtained simultaneously to build a high-quality training sample set.

[0042] S202. Use a visual large language model to reason about the image data and obtain an image reasoning description.

[0043] Specifically, a visual big language model is used to perform semantic parsing on each image in the sample, generating image reasoning descriptions that cover visual details, material features, scene associations, and other information, providing rich visual semantic supplements for text encoding.

[0044] S203. Freeze the parameters of the pre-trained text encoder, and take the image reasoning description, image and text data and corresponding target business information as input to the text encoder. After the text encoder calculates, the text features are obtained.

[0045] Specifically, a pre-trained text encoding module based on the CLIP architecture is selected as the text encoder. Its parameters, already trained on large-scale data, are frozen to prevent subsequent training from compromising its general semantic understanding capabilities. Then, the image reasoning descriptions of the samples, the original text data, and standard target business information are input into this text encoder. Through its cross-modal semantic encoding capabilities, multi-source text information is transformed into text feature vectors with unified dimensions and coherent semantics, ensuring that the features comprehensively cover the business attribute association information. S204. Using text features as input and target business information as labels, train a dimensionality reduction decoding model; wherein, the dimensionality reduction decoding model is composed of a linear projector and a text decoder connected in series.

[0046] Specifically, the generated text feature vector is used as input, and the standard target business information is used as the supervision label to train a dimensionality reduction decoding model composed of a linear projector and a text decoder. The linear projector is responsible for mapping high-dimensional text features to a low-dimensional feature space that adapts to the target business attributes. The text decoder then performs attribute prediction learning based on these low-dimensional features. The model parameters are optimized through multiple rounds of iteration until the error between the model's output business attribute prediction result and the standard label reaches a preset threshold, thus completing the training of the dimensionality reduction decoding model.

[0047] In one implementation, for furniture product attribute extraction scenarios, during training, 1000 sets of image and text data of furniture products and corresponding standard attribute labels are first collected; then, a visual large language model is used to generate inference descriptions for each set of images; subsequently, the CLIP text encoder parameters are frozen, and the inference description of the image, the parameter text of the product, the title, the details description, and the standard attribute labels are input into the encoder to generate text features of a unified dimension; finally, using the text features as input and the standard attribute labels as labels, a dimensionality reduction decoding model composed of a linear projector and a text decoder is trained, ultimately enabling the model to accurately learn the correlation rules between furniture images, text information, and attributes such as material and style. After training, it can be directly connected to the CLIP visual encoder to achieve rapid conversion of furniture images into preliminary attribute information.

[0048] Polishing prompts are used to guide the large language model to perform accuracy verification, redundant information removal, and format standardization on the initial target business information.

[0049] Specifically, the polishing prompts guide the large language model to accurately optimize the initial target business information by clarifying the instruction dimensions and execution rules. The prompts must include three core instructions: first, accuracy verification, requiring the model to cross-compare the initial target business information, image reasoning descriptions, and original text data to identify and correct issues such as contradictory attribute values ​​and semantic deviations; second, redundant information removal, clearly defining the scope of target business attributes and requiring the model to filter descriptive content and repetitive information unrelated to core attributes; and third, format standardization, specifying a unified output format to ensure that the results are structured, easy to read, and directly compatible with downstream applications.

[0050] In one implementation, the refinement prompt is set as follows: "Based on the initial furniture attribute information, image-based descriptive text, and product text data, the following processing is performed: First, the accuracy of the values ​​of core attributes such as material, style, type, form, function, color, and room classification is verified. If information from different data sources conflicts, the information with the highest visual feature matching degree is used. Second, redundant content unrelated to core attributes, such as promotional information and maintenance suggestions, is removed. Finally, the output is divided into points according to the 'attribute: value' format to ensure that no key attributes are omitted, no contradictory information is provided, and the format is consistent." Under the guidance of this prompt, the large language model can accurately complete the optimization processing of the initial information and output structured results that meet business standards.

[0051] This application also provides a zero-sample image and text information extraction system, such as... Figure 3 As shown, it includes: The data acquisition module is used to collect image and text data; image and text data includes image data and corresponding text data, wherein the text data includes the text corresponding to the image, title and description; The preliminary information extraction module is used to input image data into the business reasoning model to obtain preliminary target business information; the business reasoning model includes a visual encoder and a trained dimensionality reduction decoding model; The image description generation module is used to input image data into the visual big language model for reasoning and obtain image reasoning descriptions. The information optimization and generation module is used to generate the final target business information by taking preliminary target business information, image reasoning descriptions and text data as inputs through a large language model and combining them with polishing prompts.

[0052] The system also includes a model training module, which is used to train the dimensionality reduction decoding model, including: The sample processing unit is used to collect sample image and text data and obtain corresponding target business information. It uses a visual big language model to reason about the sample image data to obtain a sample image reasoning description. The feature extraction unit is used to select a pre-trained text encoder, freeze its parameters, and input the sample image inference description, sample text data, and target business information into the text encoder to obtain text features. The model training unit is used to train a dimensionality reduction decoding model consisting of a linear projector and a text decoder connected in series, using text features as input and target business information as labels.

[0053] The present invention further provides an electronic device, the electronic device comprising: One or more processors; Memory, used to store one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement a zero-sample image and text information extraction method.

[0054] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the zero-sample image and text information extraction method of the above embodiments.

[0055] The above embodiments are merely illustrative of the technical concept and features of the present invention, intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and should not be construed as limiting the scope of protection of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and thus all changes falling within the meaning and scope of the equivalents of the claims are intended to be included within the present invention.

Claims

1. A zero-shot document information extraction method, characterized in that, include: Collect image and text data; the image and text data includes image data and corresponding text data, wherein the text data includes the text corresponding to the image, title and description; The image data is input into the business reasoning model to obtain preliminary target business information; Image data is input into a visual large language model for reasoning to obtain an image reasoning description; The large language model takes preliminary target business information, image reasoning descriptions, and text data as input, and combines them with polishing prompts to generate the final target business information.

2. The zero-shot image-text information extraction method of claim 1, wherein, The business reasoning model includes a pre-trained visual encoder and a trained dimensionality reduction decoding model.

3. The zero-sample image and text information extraction method according to claim 2, characterized in that, The pre-trained visual encoder is a pre-trained visual encoding module based on the CLIP architecture, used to receive image data and extract high-dimensional visual features; the high-dimensional visual features include the image's texture, shape, color, target layout, and scene association information.

4. The zero-sample image and text information extraction method according to claim 2, characterized in that, The training process of the trained dimensionality reduction decoding model includes: Collect text and image data and obtain corresponding target business information; A visual large language model is used to reason about image data to obtain image reasoning descriptions; The parameters of the pre-trained text encoder are frozen, and the image reasoning description, image and text data and corresponding target business information are used as input to the text encoder. After the text encoder calculates, the text features are obtained. Using text features as input and target business information as labels, a dimensionality reduction decoding model is trained; whereby... The dimensionality reduction decoding model consists of a linear projector and a text decoder connected in series.

5. The zero-sample image and text information extraction method according to claim 4, characterized in that, The pre-trained text encoder is a pre-trained text encoding module based on the CLIP architecture. Its parameters are frozen during the training phase. It is used to receive image reasoning descriptions, image and text data and corresponding target business information, and to calculate text features through semantic encoding.

6. The zero-sample image and text information extraction method according to claim 1, characterized in that, The polishing prompts are used to guide the large language model to perform accuracy verification, redundant information removal, and format standardization processing on the initial target business information.

7. A zero-sample image and text information extraction system, characterized in that, include: The data acquisition module is used to collect image and text data; the image and text data includes image data and corresponding text data, wherein the text data includes the text corresponding to the image, title and description; The preliminary information extraction module is used to input image data into the business reasoning model to obtain preliminary target business information; the business reasoning model includes a visual encoder and a trained dimensionality reduction decoding model. The image description generation module is used to input image data into the visual big language model for reasoning and obtain image reasoning descriptions. The information optimization and generation module is used to generate the final target business information by taking preliminary target business information, image reasoning descriptions and text data as inputs through a large language model and combining them with polishing prompts.

8. A zero-sample image and text information extraction system according to claim 7, characterized in that, It also includes a model training module, which is used to train the dimensionality reduction decoding model, including: The sample processing unit is used to collect sample image and text data and obtain corresponding target business information. It uses a visual big language model to reason about the sample image data to obtain a sample image reasoning description. The feature extraction unit is used to select a pre-trained text encoder, freeze its parameters, and input the sample image inference description, sample text data, and target business information into the text encoder to obtain text features. The model training unit is used to train a dimensionality reduction decoding model consisting of a linear projector and a text decoder connected in series, using text features as input and target business information as labels.

9. An electronic device, characterized in that: The electronic device includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the zero-sample image and text information extraction method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the zero-sample image and text information extraction method as described in any one of claims 1-6.

Images